SYSTEM AND METHOD FOR ENABLING IMAGE RECOGNITION AND SEARCHING OF IMAGES

• US 20080152231A1
• Filed: 11/07/2007
• Published: 06/26/2008
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.), Palm, Inc. (TCL Technology Group Corp.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Vanhoucke, Vincent, Khan, Azhar, Yang, Danny, Lin, Xiaofan, Sumengen, Baris, Dalal, Navneet
• Granted Patent: US 7,657,100 B2
• Time in Patent Office: Days
• US Class Current: 382/209
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06Q 30/08 Auctions
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: Y10S 707/915 Image

Abstract

Embodiments programmatically analyze each of a plurality of images in order to determine one or more visual characteristics about an item shown in each of the plurality of images. Data is stored corresponding to the one or more visual characteristics. An interface in is provided for which a user is able to specify one or more search criteria. In response to receiving the one or more search criteria, a search operation is performed to identify one or more items that have a visual characteristic that satisfies at least some of the one or more search criteria.

Claims

1. A method for enabling searching of visual characteristics in images, the method:

comprising;

programmatically analyzing each of a plurality of images in order to determine one or more visual characteristics about an item shown in each of the plurality of images;

storing data corresponding to the one or more visual characteristics;

providing an interface in which a user is able to specify one or more search criteria; and

in response to receiving the one or more search criteria, performing a search operation to identify one or more items that have a visual characteristic that satisfies at least some of the one or more search criteria.

2. The method of claim 1, further comprising displaying a search result to the user that includes the identified one or more items.

3. The method of claim 1, wherein providing an interface in which a user is able to specify one or more search criteria includes enabling the user to provide a quantitative input as a search criterion.

4. The method of claim 3, wherein enabling the user to provide a quantitative input includes enabling the user to provide the quantitative input that specifies a color of a desired item.

5. The method of claim 2, wherein enabling the user to provide a quantitative input includes enabling the user to provide the quantitative input that specifies a color of a portion of an item.

6. The method of claim 5, wherein providing an interface in which a user is able to specify one or more search criteria includes enabling the user to specify search criteria that includes a combination of the specified image and one or more features that are selected from a group of features consisting of a color, a shape and a pattern.

7. The method of claim 6, wherein enabling the user to specify search criteria that includes a combination of the specified image and one or more features includes enabling the user to quantitatively specify one or more features from the group to be compared against a presence of the specified one or more features in the specified image.

8. The method of claim 2, wherein providing an interface in which a user is able to specify one or more search criteria includes enabling the user to specify an image from a previous search result as a search criterion.

9. The method of claim 1, wherein enabling the interface in which a user is able to specify one or more search criteria includes enabling the user to specify a search criterion for any one of a plurality of visual characteristics.

10. The method of claim 1, wherein enabling the interface in which a user is able to specify one or more search criteria includes enabling a user to specify a color of the item.

11. The method of claim 1, wherein enabling the interface in which a user is able to specify one or more search criteria includes enabling a user to specify a shape, a pattern, or a feature of the item.

12. The method of claim 1, further comprising storing the information that is determined from programmatically analyzing each of the plurality of images as a quantitative value.

13. The method of claim 12, wherein the quantitative value corresponds to one or more signatures that are descriptive or identifying of an individual item.

14. The method of claim 13, further comprising indexing the one or more signatures for each of the plurality of images.

15. The method of claim 1, wherein storing data corresponding to the one or more visual characteristics includes storing text terms that are descriptive of the one or more visual characteristics, the text terms being determined from programmatically analyzing the individual images.

16. The method of claim 1, wherein programmatically analyzing each of the plurality of images in order to determine information about that item includes identifying a quantitative value from analyzing individual images in the plurality of images, and converting the quantitative value into one or more text terms that are descriptive of a global or local feature of the individual images.

17. The method of claim 1, further comprising procuring at least some of the plurality of images by crawling one or more network sites.

18. The method of claim 1, wherein performing a search operation to identify one or more items includes identifying a plurality of items that have the visual characteristic that satisfies a threshold defined by the one or more search criteria, and wherein the method further comprises presenting at least some of the plurality of items that satisfy the threshold to the user, wherein presenting the at least some of the plurality of items includes presenting one or more of the plurality of items in prominence based at least in part on a determination that the one or more of the plurality of items better satisfy the threshold than other items of the plurality of items.

19. The method of claim 18, wherein providing an interface includes enabling the user to specify an image of an item as the search criteria, and wherein presenting the one or more of the plurality of items in prominence is based on the determination that the one or more of the plurality of items more resemble an appearance of the specific image of the search criteria.

20. The method of claim 19, wherein the one or more search criteria corresponds to a signature that represents one or more features of the image of the item specified by the user as the search criteria, and wherein determining that the one or more of the plurality of items most resemble an appearance of the image includes comparing the one or more signatures of the item specified by the search criteria with the stored data corresponding to the one or more visual characteristics.

21. The method of claim 19, wherein presenting the one or more of the plurality of items in prominence includes ranking multiple items based on determining a degree of resemblance between each of the multiple items and the search criteria.

22. The method of claim 19, enabling the user to specify the one or more search criteria further comprises enabling the user to specify one or more parameters that correspond to a color, a shape, or a pattern, and wherein identifying a plurality of item that have the visual characteristic that satisfies a threshold includes defining the threshold based on a combination of a signature of the image of the item and the specified one or more parameters.

23. The method of claim 18, wherein providing an interface includes enabling the user to select a portion of an image that shows a corresponding portion of an object as input.

24. The method of claim 23, further comprising comparing image data of the one or more criteria, as determined from the corresponding portion of the object, to image data from the portion of the objects shown in the plurality of images that comprise the collection, in order to determine whether the image data of individual objects in the plurality of images satisfies the threshold.

25. A system for enabling searching of images, the system comprising:

an image analysis module that analyzes individual images in a collection of images in order to determine one or more visual characteristics of an object shown in each of the individual images;

one or more memory mediums that store data corresponding to (i) each analyzed image, and (ii) each of the one or more determined visual characteristics for individual objects shown in each analyzed image; and

a search module configured to perform a search operation on the data store by comparing a search criterion to the visual characteristics of each analyzed image.

26. The system of claim 25, further comprising a user-interface that is configured to be operable by a user in order to enable the user to provide an input that corresponds to the search criterion.

27. The system of claim 25, further comprising an image procurement module that is configured to procure images in the collection, and metadata about individual images in the collection, from a plurality of network sites.

28. The system of claim 25, wherein the search module is configured to perform a search operation that identifies, as a result of the search operation, a set of images that each include a visual characteristic that is within a range that is determined by the search criterion.

29. The system of claim 25, wherein the search module is configured to perform a search operation that identifies, as a result of the search operation, a set of images that each include a visual characteristic that is deemed to be a best match for the search criterion.

30. The system of claim 25, wherein the user-interface and the search module combine to enable the user to perform a search that specifies at least one of the one or more visual characteristics quantitatively.

31. The system of claim 25, wherein the user-interface and the search module combine to enable the user to perform a search that specifies at least one of the one or more visual characteristics non-textually.

32. The system of claim 25, wherein the user-interface and the search module combine to enable the user to perform a search that specifies one of a color, shape or pattern.

33. The system of claim 25, wherein the user-interface and the search module combine to enable the user to perform the search by specifying any one or more of the color, shape, or pattern quantitatively.

34. The system of claim 25, wherein the user-interface and the search module combine to enable the user to perform the search by specifying any one or more of the color, shape, or pattern quantitatively in reference to another image that shows a specified object or portion thereof.

35. The system of claim 25, enables the user to enter input corresponding to a color.

36. The system of claim 25, wherein the image analysis module is configured to analyze individual images to determine a classification about one or more objects shown in each of the images in the collection.

37. The system of claim 25, wherein the image analysis module is configured to analyze individual images to determine a global feature about one or more objects shown in each of the images in the collection.

38. The system of claim 37, wherein the global feature corresponds to one or more of a color, a shape, metadata, or a pattern of each of the one or more objects shown in the individual images of the collection.

39. The system of claim 25, wherein the image analysis module is configured to analyze individual images to determine a local feature about one or more objects shown in each of the images in the collection.

40. The system of claim 37, wherein the local feature corresponds to one or more of a color, a shape, or a pattern of a portion each of the one or more objects shown in the individual images of the collection.

41. A system for providing a user-interface to enable image searching, the system comprising:

one or more memory mediums that store (i) a plurality of images, wherein each image shows one or more objects;

(ii) image data that identifies at least a portion of each of the one or more objects by class or uniquely; and

a user-interface module configured to enable a user to specify one or more search criteria through at least non-textual input;

a search module coupled to the user-interface module in order to compare the one or more criteria against the image data stored in the one or more medium in order to identify a search result comprising at least a portion of each image that shows an object that satisfies the search criteria.

42. The system of claim 41, wherein the image data stored in the one or more medium corresponds to one or more signatures that represent individual objects shown in the plurality of images, or portions thereof.

43. The system of claim 41, wherein the user-interface module is configured to enable the user to specify the non-textual input and a text input.

44. The system of claim 41, wherein the user-interface module is configured to enable the user to specify the non-textual input that corresponds at least in part to an image selection.

45. The system of claim 41, wherein the user-interface module is configured to enable the user to specify the non-textual input that corresponds at least in part to a quantitative specification of a color.

46. The system of claim 41, wherein the user-interface module is configured to enable the user to specify the non-textual input that corresponds at least in part to a quantitative specification of a shape or a pattern.

47. The system of claim 41, wherein the user-interface module is configured to enable the user to specify the non-textual input that corresponds at least in part to a quantitative specification of at least two attributes selected from a group consisting of a color, a shape, or a pattern.

48. The system of any of claims 45, wherein the user-interface module includes one or more sliders to enable the user to enter the non-textual input as the search criteria.

49. The system of any of claims 45, wherein the user-interface module is configured to provide a slider to enable the user to adjust the search criteria with a value that is specified by graphic movement of the slider.

50. The system of claim 49, wherein the user-interface module is configured provide the slider, and to communicate with the search module to adjust the search result based on the value specified by the slider.

51. The system of claim 49, wherein the user-interface module is configured to provide the slider, and wherein the user-interface module is configured to adjust the search result immediately in response to receiving the value from movement of the slider.

52. The system of any of claims 41, wherein the user-interface module includes at least a first feature corresponding to a picker that enables the user to enter the non-textual input.

53. The system of claim 41, wherein the user-interface module is configured to enable the user to specify the non-textual input that corresponds at least in part to a quantitative specification of a shape, a color, or a pattern.

54. The system of claim 53, wherein the user-interface is configured to enable the user to specify the non-textual information that corresponds at least in part to an image selection.

55. The system of claim 53, wherein the image selection is selected from a just previous search result that utilizes the one or more memory mediums.

56. The system of claim 53, wherein the user-interface is configured to enable the user to specify text as the one or more search criteria.