System and method for enabling image searching using manual enrichment, classification, and/or segmentation

• US 9,082,162 B2
• Filed: 09/14/2012
• Issued: 07/14/2015
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gokturk, Salih Burak, Sumengen, Baris, Vu, Diem, Dalal, Navneet, Yang, Danny, Lin, Xiaofan, Khan, Azhar, Shah, Munjal, Anguelov, Dragomir, Torresani, Lorenzo, Vanhoucke, Vincent
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US13/619,069
• Publication Number:
  US 20130127893A1
• Time in Patent Office: 1,033 Days
• Field of Search: 382/118, 382/190, 382/224, 382/173
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes

Abstract

A system includes an image analysis module that is configured to programmatically analyze individual images in a collection of images in order to determine information about each image in the collection. The system may also include a manual interface that is configured to (i) interface with one or more human editors, and (ii) displays a plurality of panels concurrently. Individual panels may be provided for one or more analyzed images, and individual panels may be configured to display information that is at least indicative of the one or more images of that panel and/or of the information determined from the one or more images.

Claims

1. A system of one or more computers configured to perform operations comprising:

programmatically analyzing individual images in a collection of images in order to determine information about each image in the collection, including segmenting each image into a respective foreground image and a respective background image;

causing a plurality of panels to be displayed concurrently to each of a plurality of human editors, each panel being provided for one or more analyzed images, and wherein each panel is configured to display a respective foreground image and a respective background image corresponding to the foreground image; and

receiving editing information from at least a first human editor to approve or reject the segmentation of a first image having a first foreground image and a first background image displayed in a first panel.

2. The system of claim 1, wherein programmatically analyzing each image in the collection comprises detecting an object in each of the images in the collection.

3. The system of claim 2, wherein the plurality of panels each display, as the information, at least a portion of each detected object.

4. The system of claim 1, wherein programmatically analyzing each image in the collection comprises a classification for an object of each image in the collection.

5. The system of claim 4, wherein the plurality of panels each display, as the information, an identification of the classification for the object of each image in the collection.

6. The system of claim 1, wherein programmatically analyzing each image in the collection comprises extracting a feature for an object of each image.

7. The system of claim 6, wherein the extracted feature corresponds to one of a color, a shape, a pattern, or a texture of the object or a portion thereof.

8. The system of claim 6, wherein the plurality of panels each display, as the information, an identification of the extracted feature for each image in the collection.

9. The system of claim 1, wherein each panel displays an editor interface configured to enable the corresponding human editor to approve or reject each foreground image and corresponding background image, and wherein the editor interface displays a Boolean style feature, carrying a default setting of approval, from which the corresponding human editor can manually toggle to a setting of rejection.

10. A method performed by a system of one or more computers, the method comprising:

programmatically analyzing individual images in a collection of images in order to determine information about each image in the collection, including segmenting each image into a respective foreground image and a respective background image;

causing a plurality of panels to be displayed concurrently to each of a plurality of human editors, each panel being provided for one or more analyzed images, and wherein each panel is configured to display a respective foreground image and a respective background image corresponding to the foreground image; and

receiving editing information from at least a first human editor to approve or reject the segmentation of a first image having a first foreground image and a first background image displayed in a first panel.

11. The method of claim 10, wherein programmatically analyzing each image in the collection comprises detecting an object in each of the images in the collection.

12. The method of claim 11, wherein the plurality of panels each display, as the information, at least a portion of each detected object.

13. The method of claim 10, wherein programmatically analyzing each image in the collection comprises a classification for an object of each image in the collection.

14. The method of claim 13, wherein the plurality of panels each display, as the information, an identification of the classification for the object of each image in the collection.

15. The method of claim 10, wherein programmatically analyzing each image in the collection comprises extracting a feature for an object of each image.

16. The method of claim 15, wherein the extracted feature corresponds to one of a color, a shape, a pattern, or a texture of the object or a portion thereof

17. The method of claim 15, wherein the plurality of panels each display, as the information, an identification of the extracted feature for each image in the collection.

18. The method of claim 10, wherein each panel displays an editor interface configured to enable the corresponding human editor to approve or reject each foreground image and corresponding background image, and wherein the editor interface displays a Boolean style feature, carrying a default setting of approval, from which the corresponding human editor can manually toggle to a setting of rejection.