SYSTEM AND METHOD FOR SEARCH PORTIONS OF OBJECTS IN IMAGES AND FEATURES THEREOF

• US 20080212899A1
• Filed: 11/07/2007
• Published: 09/04/2008
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Torresani, Lorenzo, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Vanhoucke, Vincent, Khan, Azhar, Yang, Danny, Lin, Xiaofan, Sumengen, Baris, Dalal, Navneet
• Granted Patent: US 7,657,126 B2
• Time in Patent Office: Days
• US Class Current: 382/305
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06Q 30/0603 Catalogue ordering
• CPC Class Codes: Y10S 707/949 Inventory

Abstract

Embodiments enable searching of portions of objects in images, including programmatically analyzing each image in a collection in order to determine image data that, for individual images in the collection, represents one or more visual characteristics of a portion of an object shown in that image. A user is enabled to specify one or more search criteria that includes image data, and a search result may be determined based on one or more images in the collection that show a corresponding object that has a portion that satisfies a threshold. The threshold is defined at least in part by the one or more search criteria.

Claims

1. a method for enabling searching of portions of objects in images, the method:

comprising;

programmatically analyzing each image in a collection in order to determine image data that, for individual images in the collection, represents one or more visual characteristics of a portion of an object shown in that image;

enabling a user to specify one or more search criteria that includes image data;

determining a search result based on one or more images in the collection that show a corresponding object that has a portion that satisfies a threshold, wherein the threshold is defined at least in part by the one or more search criteria.

2. The method of claim 1, wherein enabling a user to specify one or more search criteria includes enabling the user to select an image that shows a corresponding object as input.

3. The method of claim 1, wherein enabling a user to specify one or more search criteria includes enabling the user to select a portion of an image that shows a corresponding portion of an object as input.

4. The method of claim 3, wherein determining a search result based on one or more images in the collection includes comparing image data of the one or more criteria, as determined from the corresponding portion of the object, to image data from the portion of the objects shown in the plurality of images that comprise the collection, in order to determine whether the image data of any object in the plurality of images, if any, satisfies the threshold.

5. The method of claim 1, wherein enabling a user to specify one or more search criteria includes enabling the user to select (i) a portion of an image that shows a corresponding portion of an object as input, and (ii) a parameter that corresponds to a local feature of that portion.

6. The method of claim 5, wherein determining a search result based on one or more images in the collection includes comparing image data of the one or more criteria, as determined from the local features of the selected portion of the image that that shows the corresponding portion of an object as input, to image data from the portion of the objects shown in the plurality of images that comprise the collection, in order to determine whether the image data of any object in the plurality of images, if any, satisfies the threshold.

7. The method of claim 6, wherein the image data of the one or more criteria corresponds to a combination of a signature of the selected portion and the parameter of the local feature.

8. The method of claim 5, wherein the threshold corresponds to a greatest likeness, and wherein determining a search result based on one or more images in the collection includes comparing the image data of one or more criteria, as determined from the selected portion and the parameter of the local feature, to image data from the plurality of images that comprise the collection, in order to determine which of the objects in the plurality of images have the greatest likeness to the parameter of the local feature.

9. The method of claim 8, wherein the image data of the one or more criteria corresponds to a combination of a signature of the selected portion and the parameter of the local feature.

10. The method of claim 5, wherein the one or more criteria corresponds to a combination of image data determined from the portion of the selected image, as weighted by the selected parameter of the local feature.

11. The method of claim 10, wherein the local feature corresponds to color, and wherein the parameter specifies a hue.

12. The method of claim 10, wherein the local feature corresponds to a shape or a pattern, and wherein the parameter specifies a class or particular shape or pattern.

13. The method of claim 10, wherein the local feature corresponds to anyone of color, shape, or pattern, and wherein the method further comprises enabling the user to specify a parameter for any and all of the local features, and wherein image data from each specified parameter weights the image data of the one or more criteria.

14. A method for enabling searching of images based on image features, the method comprising:

programmatically analyzing each image in a collection in order to determine image data that, for individual images in the collection, represents one or more features of at least a portion of the image;

enabling a user to specify one or more search criteria that includes features;

determining a search result based on one or more images in the collection that show a corresponding object that has a portion that satisfies a threshold, wherein the threshold is defined at least in part by the one or more search criteria.

15. The method of claim 14, wherein enabling a user to specify one or more search criteria includes enabling the user to select a parameter that corresponds to a global feature of an object with desired characteristics, and wherein determining the search result includes determining image data that corresponds to the parameter.

16. The method of claim 15, wherein the global feature corresponds to color, and wherein the parameter specifies a hue.

17. The method of claim 15, wherein the global feature corresponds to a shape or a pattern, and wherein the parameter specifies a class or particular shape or pattern.

18. The method of claim 15, wherein determining a search result based on one or more images in the collection includes comparing the image data of one or more criteria, as determined from the selected parameter, to image data from the plurality of images that comprise the collection, in order to determine which of the objects in the plurality of images are most similar in appearance to the parameter of the global feature.

19. The method of claim 15, wherein enabling a user to specify one or more search criteria includes enabling the user to select a given image of the object, and wherein enabling the user to select a parameter includes enabling the user to select the parameter that corresponds to a given global feature of that object, wherein the parameter weights how the global feature is represented in the image data of the one or more search criteria.

20. The method of claim 19, wherein determining the search result includes determining image data that corresponds to the image data of the one or more criteria, as weighted by the parameter of the global feature.

21. The method of claim 20, wherein the global feature corresponds to color, and wherein the parameter specifies a hue.

22. The method of claim 20, wherein the global feature corresponds to a shape or a pattern, and wherein the parameter specifies a class or particular shape or pattern.

23. The method of claim 20, wherein the global feature corresponds to anyone of color, shape, or pattern, and wherein the method further comprises enabling the user to specify a parameter for any and all of the global features, and wherein image data from each specified parameter weights the image data of the one or more criteria.

24. The method of claim 15, wherein enabling a user to specify one or more search criteria includes enabling the user to select a given image of the object, and wherein enabling the user to select a parameter includes enabling the user to select the parameter that corresponds to a given global feature of that object, wherein the parameter weights how the global feature is represented in the image data of the one or more search criteria.

25. The method of claim 24, wherein determining the search result includes determining image data that corresponds to the image data of the one or more criteria, as weighted by the parameter of the global feature.

26. The method of claim 24, wherein the global feature corresponds to color, and wherein the parameter specifies a hue.

27. The method of claim 24, wherein the global feature corresponds to a shape or a pattern, and wherein the parameter specifies a class or particular shape or pattern.

28. The method of claim 24, wherein the global feature corresponds to anyone of color, shape, or pattern, and wherein the method further comprises enabling the user to specify a parameter for any and all of the global features, and wherein image data from each specified parameter weights the image data of the one or more criteria.

29. The method of claim 28, wherein the method includes enabling the user to specify the parameter for one or more of the global features by operating a graphical feature that is in form of a slider that can be moved in between a range of values for each of the one or more global features.

30. A method for enabling searching of images using different types of criteria, the method comprising:

obtaining an image of each of a plurality of items;

programmatically analyzing the image of each of the plurality of items in order to determine one or more visual characteristics of (i) the item, or (ii) a feature or portion of the item;

storing the visual characteristic determined from each image;

enabling an interface in which a user is able to specify a search criterion that includes an image, wherein the search criterion is of any one of multiple types; and

in response to receiving the search criterion, performing a search of the data store by referencing the search criterion against the stored visual characteristics.

31. The method of claim 30, wherein programmatically analyzing the image of each of the plurality of items in order to determine one or more visual characteristics includes determining, from the image of each of the plurality of items, one or more characteristics of (i) a signature of the item or a portion thereof, (ii) one or more primary colors of the item or the portion thereof, (iii) a shape of the item or the portion thereof, (iv) a pattern of the item or the portion thereof.

32. The method of claim 30, wherein enabling an interface in which a user is able to specify a search criterion that includes an image includes determining a recognition signature of the image of the search criterion, and wherein performing a search operation includes performing a comparison using the image of the search criterion and the signature of at least some of the recognition signatures stored in the data store.

33. The method of claim 30, wherein enabling an interface in which a user is able to specify a search criterion that includes an image includes enabling the user to select an entry image from the data store.