SYSTEM AND METHOD FOR ENABLING IMAGE SEARCHING USING MANUAL ENRICHMENT, CLASSIFICATION, AND/OR SEGMENTATION

• US 20080144943A1
• Filed: 11/07/2007
• Published: 06/19/2008
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Torresani, Lorenzo, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Vanhoucke, Vincent, Khan, Azhar, Yang, Danny, Lin, Xiaofan, Sumengen, Baris, Dalal, Navneet
• Granted Patent: US 7,660,468 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes

Abstract

Embodiments described herein provide for a system for creating a data collection of recognized images. The system includes an image analysis module that is configured to programmatically analyze individual images in a collection of images in order to determine information about each image in the collection. The system may also include a manual interface that is configured to (i) interface with one or more human editors, and (ii) displays a plurality of panels concurrently. Individual panels may be provided for one or more analyzed images, and individual panels may be configured to display information that is at least indicative of the one or more images of that panel and/or of the information determined from the one or more images. Additionally, the manual interface enables the one or more human editors to view the plurality of panels concurrently and to interact with each of the plurality of panels in order to correct or remove any information that is incorrectly determined from the image of that panel.

Claims

1. A system for creating a data collection of recognized images, the system comprising:

an image analysis module that is configured to programmatically analyze individual images in a collection of images in order to determine information about each image in the collection;

a manual interface that is configured to (i) interface with one or more human editors, and (ii) displays a plurality of panels concurrently, each panel being provided for one or more analyzed images, and wherein each panel is configured to display information that is at least indicative of the one or more images of that panel and/or of the information determined from the one or more images;

wherein the manual interface enables the one or more human editors to view the plurality of panels concurrently and to interact with each of the plurality of panels in order to correct or remove any information that is incorrectly determined from the image of that panel.

2. The system of claim 1, wherein the image analysis module is configured to programmatically analyze each image in the collection in order to detect an object in each of the images in the collection.

3. The system of claim 2, wherein the plurality of panels each display, as the information, at least a portion of each detected object.

4. The system of claim 1, wherein the image analysis module is configured to programmatically analyze each image in the collection in order to determine a classification for an object of each image in the collection.

5. The system of claim 2, wherein the plurality of panels each display, as the information, an identification of the classification for the object of each image in the collection.

6. The system of claim 1, wherein the image analysis module is configured to programmatically analyze each image in the collection in order to extract a feature for an object of each image.

7. The system of claim 6, wherein the extracted feature corresponds to one of a color, a shape, a pattern, or a texture of the object or a portion thereof.

8. The system of claim 5, wherein the plurality of panels each display, as the information, an identification of the extracted feature for each image in the collection.

9. The system of claim 1, wherein the manual interface displays a plurality of panels in cascade, wherein each panel displays the classification of the object shown in that panel, and wherein the editor interface is configured to enable the editor to deselect or correct an incorrect information determined from the image of each panel.

10. A computer-implemented method for processing images for search or analysis, the method comprising:

retrieving or receiving an image from an external source;

identifying an object of the image by separating a foreground of the image from a background of the image.

11. The method of claim 10, further comprising classifying the object of the image.

12. The method of claim 10, wherein classifying the object includes using text and/or metadata associated with the image to determine a classification of the object.

13. The method of claim 10, wherein the external source corresponds to a webpage at a third-party site.

14. The method of claim 10, wherein the external source corresponds to a webpage at a third-party e-commerce site, and wherein classifying the object of the image includes determining that the object is a type of merchandise.

15. The method of claim 10, further comprising extracting one or more visual features from the object in the image.

16. The method of claim 15, wherein the one or more features correspond to a global feature.

17. The method of claim 15, wherein the one or more features correspond to a local feature.

18. The method of claim 15, wherein the one or more features correspond to one or more of (i) a global color of the object, or (ii) a localized color of a portion of the object.

19. The method of claim 15, wherein the one or more features correspond to a texture, a pattern, or a shape of the object.

20. The method of claim 10, further comprising programmatically determining information about the object of the image from analyzing one or more of the image or metadata associated with the image, and indexing the information.

21. A method for searching a data collection of images, the method comprising the steps of:

a) determining a search criteria that includes a signature of image data;

b) identifying a first portion of the search criteria, the first portion of the search criteria corresponding to a first portion of the signature;

c) searching the data collection for individual images that satisfy the first portion of the search criteria, in order to identify a first subset of the data collection;

d) identifying at least another portion of the search criteria, including another portion of the signature, the other portion of the search criteria being more specific than the first portion;

e) searching the first subset of the data collection using the at least other portion of the search criteria.

22. The method of claim 21, further comprising repeating steps d) and e) so as to progressively make the search criteria more specific while identifying a smaller subset of the search criteria.

23. A system for creating a data collection of recognized images, the system comprising:

an image analysis subsystem that is configured to programmatically analyze individual images in a collection of images;

wherein the image analysis subsystem includes a first component to segment an object from each image in the collection.

24. The system of claim 23, wherein the image analysis subsystem further comprises a second component to classify the object that is segmented from each image.

25. The system of claim 24, wherein the second component uses text and/or metadata that is provided or associated with each image to classify the object that is segmented from that image.

26. The system of claim 25, wherein the image analysis subsystem further comprises a third component to extract one or more of a global or a local feature from the object that is segmented from each image.

27. The system of claim 26, wherein the one or more features correspond to one or more of (i) a global color of the object, or (ii) a localized color of a portion of the object.

28. A system for creating a data collection of recognized images, the system comprising:

an image analysis subsystem that is configured to programmatically analyze individual images in a collection of images in order to detect one or more objects in each of the images in the collection; and

wherein the image analysis subsystem includes an object determinator to classify each of the one or more objects in each of the images into one or more categories.

29. The system of claim 28, wherein the object determinator uses text and/or metadata that is provided or associated with each image to classify the object that is shown in that image.

30. The system of claim 29, wherein the object determinator uses image data from each image to classify the object that is shown in that image.

31. The system of claim 28, further comprising a search subsystem that enables a user to search objects contained in the images in the collection by specifying a category corresponding to the one or more categories.