System and method for search portions of objects in images and features thereof

• US 8,345,982 B2
• Filed: 12/28/2009
• Issued: 01/01/2013
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gokturk, Salih Burak, Sumengen, Baris, Vu, Diem, Dalal, Navneet, Yang, Danny, Lin, Xiaofan, Khan, Azhar, Shah, Munjal, Anguelov, Dragomir, Torresani, Lorenzo, Vanhoucke, Vincent
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US12/648,287
• Publication Number:
  US 20100135582A1
• Time in Patent Office: 1,100 Days
• Field of Search: 382/190, 382/209
• US Class Current: 382/190
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06Q 30/0603 Catalogue ordering
• CPC Class Codes: Y10S 707/949 Inventory

Abstract

Embodiments enable searching of portions of objects in images, including programmatically analyzing each image in a collection in order to determine image data that, for individual images in the collection, represents one or more visual characteristics of a portion of an object shown in that image. A user is enabled to specify one or more search criteria that includes image data, and a search result may be determined based on one or more images in the collection that show a corresponding object that has a portion that satisfies a threshold. The threshold is defined at least in part by the one or more search criteria.

Claims

1. A method for enabling searching of images based on image features, the method comprising:

using one or more processors to perform steps comprising;

programmatically analyzing each image in a collection of images, including determining at least a local feature and a global feature for individual objects that appear in each image of the collection, the local feature of each object corresponding to a visual characteristic of a segment of that object, the global feature of each object corresponding to a visual characteristic of the object as a whole;

enabling a user to specify search criteria based on an input that includes a first global feature parameter and a local feature;

obtaining a first search result comprising one or more images that satisfy the search criteria;

enabling the user to modify the search criteria by specifying a second global feature parameter in addition to the first global feature parameter and the local feature; and

obtaining a second search result comprising one or more images that satisfy the modified search criteria.

2. The method of claim 1, wherein enabling the user to specify the search criteria includes enabling the user to select a parameter that corresponds to a color, a shape or a pattern.

3. The method of claim 1, wherein obtaining the first search result includes:

comparing image data, corresponding to the search criteria, to local and global features of objects depicted in at least some images of the collection,

determining, from comparing the image data, N objects in images that comprise a portion of the collection, the N objects having at least a segment that is deemed most similar in appearance to the image data.

4. The method of claim 1, wherein enabling the user to specify the search criteria includes enabling the user to select the object depicted in a given input image, and to specify the first global feature parameter by selecting or weighting a visual characteristic that corresponds to either a local or global feature of the object depicted in the given input image.

5. The method of claim 1, wherein enabling the user to specify the search criteria includes enabling the user to select an object depicted in a given input image, and to specify the first global feature parameter that corresponds to a shape or a pattern of the object in the given input image.

6. The method of claim 1, wherein enabling the user to specify the search criteria includes enabling the user to select an object depicted in a given input image and to specify the first global feature parameter that corresponds to a color value of the object depicted in the given input image.

7. The method of claim 6, wherein enabling the user to specify the first global feature parameter includes enabling the user to specify a parameter value by operating a graphical feature that is in the form of a slider that can be moved in between a range of values for a given visual characteristic.

8. The method of claim 1, wherein enabling the user to modify the search criteria includes enabling the user to vary an input corresponding to the first global feature parameter along a continuum that includes a minimum and a maximum.

9. The method of claim 8, further comprising:

receiving input that indicates the user varied a parameter; and

wherein the second search result is determined in response to receiving the input.

10. A non-transitory computer readable medium for enabling searching of images based on image features, the computer-readable medium comprising:

using one or more processors to perform steps comprising;

programmatically analyzing each image in a collection of images, including determining at least a local feature and a global feature for individual objects that appear in each image of the collection, the local feature of each object corresponding to a visual characteristic of a segment of that object, the global feature of each object corresponding to a visual characteristic of the object as a whole;

enabling a user to specify search criteria based on an input that includes a first global feature parameter and a local feature;

obtaining a first search result comprising one or more images that satisfy the search criteria;

enabling the user to modify the search criteria by specifying a second global feature parameter in addition to the first global feature parameter and the local feature; and

obtaining a second search result comprising one or more images that satisfy the modified search criteria.

11. The computer-readable medium of claim 10, wherein the instructions for enabling the user to specify the search criteria include instructions for enabling the user to select a parameter that corresponds to a color, a shape or a pattern.

12. The computer-readable medium of claim 10, wherein the instructions for obtaining a first search result include instructions for:

comparing image data, corresponding to the search criteria, to local and global features of objects depicted in at least some images of the collection,

determining, from comparing the image data, N objects in images that comprise a portion of the collection, the N objects having at least a segment that is deemed most similar in appearance to the image data.

13. The computer-readable medium of claim 10, wherein the instructions for enabling the user to specify the search criteria include instructions for enabling the user to select the object depicted in a given input image, and to specify the first global feature parameter by selecting or weighting a visual characteristic that corresponds to either a local or global feature of the object depicted in the given input image.

14. The computer-readable medium of claim 10, wherein the instructions for enabling the user to specify the search criteria include instructions for enabling the user to select an object depicted in a given input image, and to specify the first global feature parameter that corresponds to a shape or a pattern of the object in the given input image.

15. The computer-readable medium of claim 10, wherein the instructions for enabling the user to specify the search criteria include instructions for enabling the user to select an object depicted in a given input image, and to specify the first global feature parameter that corresponds to a color value of the object depicted in the given input image.

16. The computer-readable medium of claim 15, wherein the instructions for enabling the user to specify the first global feature parameter include instructions for enabling the user to specify a parameter value by operating a graphical feature that is in the form of a slider that can be moved in between a range of values for a given visual characteristic.

17. The computer-readable medium of claim 10, wherein the instructions for enabling the user to modify the search criteria includes enabling the user to vary an input corresponding to the first global feature parameter along a continuum that includes a minimum and a maximum.

18. The computer-readable medium of claim 17, further comprising instructions for receiving input that indicates the user varied a parameter;

and

wherein the instructions provide for determining the second search result in response to receiving the input.

19. A computer system comprising:

a memory that stores a set of instructions;

one or more processors that use the instructions from the memory to;

programmatically analyze each image in a collection of images, including determining at least a local feature and a global feature for individual objects that appear in each image of the collection, the local feature of each object corresponding to a visual characteristic of a segment of that object, the global feature of each object corresponding to a visual characteristic of the object as a whole;

enable a user to specify search criteria based on an input that includes a first global feature parameter and a local feature;

obtain a first search result comprising one or more images that satisfy the search criteria;

enable the user to modify the search criteria by specifying a second global feature parameter in addition to the first global feature parameter and the local feature; and

obtain a second search result comprising one or more images that satisfy the modified search criteria.

20. The computer system of claim 19, wherein the processor enables the user to specify the search criteria by selecting a parameter that corresponds to a color, a shape or a pattern.