System and method for search portions of objects in images and features thereof

• US 9,008,435 B2
• Filed: 09/14/2012
• Issued: 04/14/2015
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gokturk, Salih Burak, Sumengen, Baris, Vu, Diem, Dalal, Navneet, Yang, Danny, Lin, Xiaofan, Khan, Azhar, Shah, Munjal, Anguelov, Dragomir, Torresani, Lorenzo, Vanhoucke, Vincent
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US13/619,939
• Publication Number:
  US 20130121571A1
• Time in Patent Office: 942 Days
• Field of Search: 382/190, 382/209
• US Class Current: 382/190
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06Q 30/0603 Catalogue ordering
• CPC Class Codes: Y10S 707/949 Inventory

Abstract

Embodiments enable searching of portions of objects in images, including programmatically analyzing each image in a collection in order to determine image data that, for individual images in the collection, represents one or more visual characteristics of a portion of an object shown in that image. A user is enabled to specify one or more search criteria that includes image data, and a search result may be determined based on one or more images in the collection that show a corresponding object that has a portion that satisfies a threshold. The threshold is defined at least in part by the one or more search criteria.

Claims

1. A method for enabling searching of images based on image features, the method comprising:

using one or more processors to perform steps comprising;

programmatically analyzing each image in a collection of images, including determining at least a local feature and a global feature for individual objects that appear in each image of the collection, the local feature of each object corresponding to a visual characteristic of a segment of that object, the global feature of each object corresponding to a visual characteristic of the object as a whole, wherein determining at least a local feature and a global feature for individual objects that appear in each image of the collection comprises, for each object in the image, determining at least a first global feature independently from determining at least a first local feature and by using a global feature extraction module to determine the first global feature from the image;

enabling a user to specify a search input; and

determining a search result comprising

(i) one or more first images that depict objects that individually have a global feature which satisfies a criteria identified from the input, and

(ii) one or more second images that depict objects that individually have a local feature that satisfies the criteria.

2. The method of claim 1, wherein enabling a user to specify the search input includes enabling the user to select a parameter that corresponds to a color, a shape or a pattern.

3. The method of claim 2, wherein determining a search result includes comparing image data, corresponding to the search input, to local and global features of objects depicted in at least some images of the collection, in determining N objects in images that comprise a portion of the collection, which have at least a segment that is deemed most similar in appearance to the image data.

4. The method of claim 2, wherein enabling a user to specify the search input includes enabling the user

(i) to select an object depicted in a given image, and

(ii) to specify a parameter that selects or weights a visual characteristic that corresponds to either a local or global feature of the object in the given image.

5. The method of claim 2, wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to a shape or a pattern.

6. The method of claim 2, wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to criteria color value.

7. The method of claim 6, wherein enabling the user to specify the parameter includes enabling the user to specify a parameter value by operating a graphical feature that is in form of a slider that can be moved in between a range of values for a given visual characteristic.

8. The method of claim 1, wherein the global feature module includes at least one of a color feature extraction component, a shape feature component, or a text feature component, and

wherein determining at least a first global feature comprises determining at least one of a global color feature, a global shape feature, or a global text feature of the object in the image.

9. The method of claim 1, wherein determining at least a first local feature comprises:

identifying one or more segments of the object in the image; and

executing a local feature extraction module to determine the first local feature using the identified segments.

10. The method of claim 1, wherein determining a search result comprises:

determining the one or more first images that depict objects that individually have a global feature which satisfies a criteria identified from the input; and

determining, from the one or more first images, the one or more second images that depict objects that individually have a local feature that satisfies the criteria.

11. The method of claim 1, wherein programmatically analyzing each image in a collection of images comprises:

segmenting content of the image into a foreground image and a background image; and

segmenting the individual objects from one another in the foreground image, the individual objecting being separated from the background image.

12. A system, comprising:

a data processing apparatus; and

a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing to perform operations comprising;

programmatically analyzing each image in a collection of images, including determining at least a local feature and a global feature for individual objects that appear in each image of the collection, the local feature of each object corresponding to a visual characteristic of a segment of that object, the global feature of each object corresponding to a visual characteristic of the object as a whole, wherein determining at least a local feature and a global feature for individual objects that appear in each image of the collection comprises, for each object in the image, determining at least a first global feature independently from determining at least a first local feature and by using a global feature extraction module to determine the first global feature from the image;

enabling a user to specify a search input; and

determining a search result comprising

(i) one or more first images that depict objects that individually have a global feature which satisfies a criteria identified from the input, and

(ii) one or more second images that depict objects that individually have a a local feature that satisfies the criteria.

13. The system of claim 12, wherein enabling a user to specify the search input includes enabling the user to select a parameter that corresponds to a color, a shape or a pattern.

14. The system of claim 12, wherein determining a search result includes comparing image data, corresponding to the search input, to local and global features of objects depicted in at least some images of the collection, in determining N objects in images that comprise a portion of the collection, which have at least a segment that is deemed most similar in appearance to the image data.

15. The system of claim 12, wherein enabling a user to specify the search input includes enabling the user

(i) to select an object depicted in a given image, and

(ii) to specify a parameter that selects or weights a visual characteristic that corresponds to either a local or global feature of the object in the given image.

16. The system of claim 12, wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to a shape or a pattern.

17. The system of claim 12, wherein enabling the user to specify the search input includes enabling the user to select an object depicted in an image and a parameter that corresponds to criteria color value.

18. The system of claim 12, wherein enabling the user to specify the parameter includes enabling the user to specify a parameter value by operating a graphical feature that is in form of a slider that can be moved in between a range of values for a given visual characteristic.

19. The system of claim 12, wherein the global feature module includes at least one of a color feature extraction component, a shape feature component, or a text feature component,

wherein determining at least a first global feature comprises determining at least one of a global color feature, a global shape feature, or a global text feature of the object in the image, and

wherein determining at least a first local feature comprises:

identifying one or more segments of the object in the image; and

executing a local feature extraction module to determine the first local feature using the identified segments.

20. The system of claim 12, wherein programmatically analyzing each image in a collection of images comprises:

segmenting content of the image into a foreground image and a background image; and

segmenting the individual objects from one another in the foreground image, the individual objecting being separated from the background image.