System and method for enabling image searching using manual enrichment, classification, and/or segmentation

• US 7,660,468 B2
• Filed: 11/07/2007
• Issued: 02/09/2010
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Torresani, Lorenzo, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Vanhoucke, Vincent, Khan, Azhar, Yang, Danny, Lin, Xiaofan, Sumengen, Baris, Dalal, Navneet
• Primary Examiner(s): Kim; Charles
• Application Number:
  US11/936,713
• Publication Number:
  US 20080144943A1
• Time in Patent Office: 825 Days
• Field of Search: None
• US Class Current: 382/224
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes

Abstract

Embodiments described herein provide for a system for creating a data collection of recognized images. The system includes an image analysis module that is configured to programmatically analyze individual images in a collection of images in order to determine information about each image in the collection. The system may also include a manual interface that is configured to (i) interface with one or more human editors, and (ii) displays a plurality of panels concurrently. Individual panels may be provided for one or more analyzed images, and individual panels may be configured to display information that is at least indicative of the one or more images of that panel and/or of the information determined from the one or more images. Additionally, the manual interface enables the one or more human editors to view the plurality of panels concurrently and to interact with each of the plurality of panels in order to correct or remove any information that is incorrectly determined from the image of that panel.

Claims

1. A method for processing images, the method comprising:

using one or more processors to perform steps comprising;

analyzing an image provided at a third-party website in connection with displaying concurrently one or more other images at the third-party website that satisfy a search criterion generated from analyzing the image, wherein analyzing the image includes;

programmatically analyzing the image to determine one or more visual characteristics of at least one object in the image;

generate a search criterion based at least in part on the one or more visual characteristics of the at least one object;

performing a search of a data store using the search criterion in order to obtain a search result that includes one or more images of merchandise objects that satisfy the search criterion; and

presenting the one or more images of merchandise objects with the analyzed image at the third-party website;

wherein programmatically analyzing the image further comprises determining one or more categories of the at least one object of the analyzed image based at least in part on (i) text and/or metadata associated with the image; and

(ii) a reference record assigned to each of the one or more categories that identifies words or phrases and their respective weights for use in determining that the one or more categories are relevant to the analyzed image;

wherein performing the search further comprises identifying at least one image of a merchandise object that (i) is deemed similar to the at least one object of the analyzed image, but (ii) is in a different category than the at least one object of the analyzed image.

2. The method of claim 1, wherein determining one or more categories of the at least one object includes using text and/or metadata associated with the image to determine the one or more categories.

3. The method of claim 1, further comprising segmenting the analyzed image to cause the at least one object of the analyzed image to be segmented apart from a remainder of the analyzed image so as to be selectable by a user to generate said search criterion corresponding to the selected object.

4. The method of claim 1, wherein the third-party website corresponds to a blog.

5. The method of claim 1, wherein the one or more visual characteristics of the analyzed image include a global feature.

6. The method of claim 1, wherein the one or more visual characteristics of the analyzed image include a local feature.

7. The method of claim 1, wherein the one or more visual characteristics of the analyzed image include (i) a global color of the at least one object, or (ii) a localized color of a portion of the at least one object.

8. The method of claim 1, wherein the one or more visual characteristics of the analyzed image include a texture, a pattern, or a shape of the at least one object.

9. The method of claim 1, wherein programmatically analyzing the image includes programmatically determining information about the at least one object of the image by analyzing metadata associated with the images.

10. The method of claim 1, wherein the text and/or metadata is indicative of a source from which the at least one object of the image is provided.

11. The method of claim 1, wherein the text and/or metadata associates gender or demographic information with a source that displays the image that is analyzed.

12. The system of claim 1, wherein the text and/or metadata is indicative of a source from which the at least one object of the image is provided.

13. The system of claim 1, wherein the text and/or metadata associates gender or demographic information with a source that displays the image that is analyzed.

14. A system for processing images for analysis and search, the system comprising:

a component to procure one or more images from a third-party website;

an image analysis subsystem that is configured to;

programmatically analyze at least one of the procured images to determine one or more visual characteristics of at least one object in the image;

determine one or more categories of the at least one object of the analyzed image based at least in part on (i) text and/or metadata associated with the image; and

(ii) a reference record assigned to each of the one or more categories that identifies words or phrases and their respective weights for use in determining that the one or more categories are relevant to the analyzed image;

generate one or more search criteria from the one or more visual characteristics of the at least one object;

search the data store using the one or more search criteria to identify one or more images of merchandise objects that satisfy the one or more search criteria, including identifying at least one image of a merchandise object that (i) is deemed similar to the at least one object of the analyzed image, but (ii) is in a different category than the at least one object;

a presentation module configured to present the identified one or more images of merchandise objects with the analyzed image of the at least one object, including to present the image of the merchandise object that is of the different category than the at least one object of the analyzed image.

15. The system of claim 14, further comprising a segmentation module configured to segment the analyzed image so as to make the at least one object selectable separate from a remaining portion of that image.

16. The system of claim 15, wherein the image analysis subsystem is configured to determine a category of the at least one object of the analyzed image by using text and/or metadata that is provided or associated with the analyzed image to categorize the at least one object.

17. The system of claim 16, wherein the image analysis subsystem further includes an extraction component to extract one or more features from the at least one object of the analyzed image.

18. The system of claim 17, wherein the one or more features correspond to one or more of (i) a global color of the at least one object, or (ii) a localized color of a portion of the at least one object.