Identifying objects within an image

• US 9,754,177 B2
• Filed: 06/21/2013
• Issued: 09/05/2017
• Est. Priority Date: 06/21/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, Ce, Weiss, Yair, Torralba Barriuso, Antonio
• Primary Examiner(s): Rush, Eric
• Application Number:
  US13/923,820
• Publication Number:
  US 20140376819A1
• Time in Patent Office: 1,537 Days
• Field of Search: 382159, 382173, 382180, 382181, 382190, 382195, 382206, 382209, 382224, 382282, 382291
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/52 Scale-space analysis, e.g. wavelet analysis multi-scale boundary representations G06V10/42
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects

Abstract

One or more aspects of the subject disclosure are directed towards identifying objects within an image via image searching/matching. In one aspect, an image is processed into bounding boxes, with the bounding boxes further processed to each surround a possible object. A sub-image of pixels corresponding to the bounding box is featurized for matching with tagged database images. The information (tags) associated with any matched images is processed to identify/categorize the sub-image and thus the object corresponding thereto.

Claims

1. A computer-implemented method for processing an image, the method comprising:

creating a first bounding box around a first region of interest in an image;

creating a second bounding box around a second region of interest in the image;

creating a third bounding box around the first region of interest in the image;

determining that the first bounding box and the third bounding box meet a variance threshold;

determining that the second bounding box does not meet the variance threshold;

determining the first bounding box and the third bounding box overlap;

based on the determination that the first bounding box and the third bounding box overlap, merging the first bounding box and the third bounding box into a single fourth bounding box;

extracting a sub-image corresponding to the fourth bounding box;

querying a remote image-to-image search engine using the sub-image to find a plurality of matched images for the sub-image, the plurality of matched images comprising a first matched image, the first matched image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;

determining a first matching value based on matching the sub-image with the first matched image;

determining that the first matching value is greater than a matching value threshold;

based on determining that the first matching value is greater than the matching value threshold, labeling the sub-image with a sub-image tag that corresponds to an identity of the second object; and

outputting the image with the labeled sub-image.

2. The method of claim 1 further comprising:

segmenting the image into segments; and

creating the first and the second bounding boxes from the segments.

3. The method of claim 2 further comprising, grouping a fifth and sixth bounding box together based upon similarity.

4. The method of claim 1 further comprising extracting features from the sub-image.

5. The method of claim 4 wherein extracting the features comprises using multi-orientation steerable filters.

6. The method of claim 1, wherein labeling the sub-image comprises:

integrating the identity of the second object; and

pruning the identity of the second object to fit to the sub-image.

7. The method of claim 6, wherein pruning the identity of the second object comprises determining what identifying information of the one or more matched images is semantically coherent.

8. The method of claim 6, wherein pruning the identity of the second object comprises pruning noisy tags.

9. The method of claim 1, wherein the identity of the second object is a species of the category of the second object.

10. A system comprising:

a memory including an input image; and

a processor configured to;

create a first bounding box around a first region of interest in the input image;

create a second bounding box around a second region of interest in the input image;

create a third bounding box around the first region of interest in the input image;

determine that the first bounding box and the third bounding box meet a variance threshold;

determine that the second bounding box does not meet the variance threshold;

determine the first bounding box and the third bounding box overlap;

based on the determination that the first bounding box and the third bounding box overlap, merge the first bounding box and the third bounding box into a single fourth bounding box;

select a sub-image corresponding to the fourth bounding box;

query an image-to-image search engine using the sub-image to obtain identifying information of a plurality of matched images, the plurality of matched images comprising a first matched image, the first matched image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;

process the identifying information, the process comprising;

determining a first matching value based on matching the sub-image with the first matched image;

determining that the first matching value is greater than a matching value threshold; and

based on determining that the first matching value is greater than the matching value threshold, generating an identifier for the sub-image, the identifier corresponding to the category of the second object.

11. The system of claim 10 wherein the processor is incorporated into one or more of the following:

a search engine, a mobile computing device, a personal computing device, and a cloud service.

12. The system of claim 10 wherein the processor is further configured to output a parsed image labeled with the identifier.

13. The system of claim 10 wherein the processor is further configured to extract features from the sub-image, and to query the image-to-image search engine with at least some of the features to obtain the identifying information of the one or more matched images.

14. The system of claim 13 wherein the features comprise global features.

15. The system of claim 10, wherein selecting the sub-image corresponding to the fourth bounding box further comprises determining the fourth bounding box includes a color of interest.

16. One or more computer-readable hardware storage devices storing computer executable instructions, which upon execution perform operations, comprising:

inputting an image;

creating a first bounding box around a first region of interest in the image;

creating a second bounding box around a second region of interest in the image;

creating a third bounding box around the first region of interest in the image;

determining that the first bounding box and the third bounding box meet a variance threshold;

determining that the second bounding box does not meet the variance threshold;

determining the first bounding box and the third bounding box overlap;

based on the determination that the first bounding box and the third bounding box overlap, merging the first bounding box and the third bounding box into a single fourth bounding box;

extracting a sub-image corresponding to the fourth bounding box;

determining sub-image features of the sub-image;

using the sub-image, querying an image-to-image search engine including a plurality of tagged images with features corresponding to the sub-image features of the sub-image, the plurality of tagged images comprising a first tagged image, the first tagged image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;

determining a first matching value based on matching the sub-image with the first tagged image;

determining that the first matching value is greater than a matching value threshold; and

based on determining that the first matching value is greater than the matching value threshold, matching the first object corresponding to the sub-image with the first tagged image.

17. The one or more computer-readable hardware storage devices of claim 16, wherein matching the first object comprises processing information associated with the first tagged image to categorize or identify the first object.

18. The one or more computer-readable hardware storage devices of claim 16, wherein generating the first, second, and third bounding boxes comprises segmenting the image into segments and processing the segments.

19. The one or more computer-readable hardware storage devices of claim 18, wherein processing the segments comprises determining that the fourth bounding box meets a color variance threshold.

20. The one or more computer-readable hardware storage devices of claim 16 storing further computer executable instructions, which upon execution perform an operation comprising, labeling the first object in the image that is matched to the first tagged image with an identity of the second object.