SYSTEM AND METHOD FOR ENABLING IMAGE RECOGNITION AND SEARCHING OF REMOTE CONTENT ON DISPLAY

• US 20080082426A1
• Filed: 07/13/2007
• Published: 04/03/2008
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Miller, Andrew, Moran, Mark, Gokturk, Salih, Chiao, Dan, Phillips, Jacquie, Shah, Munjal, Vu, Diem, Vanhoucke, Vincent, Khan, Azhar, Lin, Xiaofan, Dalal, Navneet
• Granted Patent: US 8,732,025 B2
• Time in Patent Office: Days
• US Class Current: 705/27
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0625 Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06Q 30/08 Auctions

Abstract

Images are analyzed by programmatic mechanisms for accessing one or more remote web pages to retrieve content on display at the remote web pages. The retrieved images may be analyzed to determine information about an object shown in a corresponding image of the content on display. At least a portion of the object shown in the corresponding image of the content on display may be made selectable and associated with the determined information. This determined information may subsequently be used, in for example, search applications.

Claims

1. A method for analyzing images, the method comprising:

accessing one or more remote web pages to retrieve content on display at the remote web pages;

analyzing the retrieved content on display to determine information about an object shown in a corresponding image of the content on display, wherein analyzing the content on display includes analyzing the corresponding image;

enabling at least a portion of the object shown in the corresponding image of the content on display to be subsequently selectable and associated with the determined information; and

responsive to detecting a selection of at least the portion of the object in the image, using the determined information.

2. The method of claim 1, further comprising receiving a trigger signal from a site of the remote web page, and wherein accessing one or more remote web pages to retrieve content on display is performed in response to receiving the trigger signal.

3. The method of claim 2, wherein the method further comprises remotely causing the trigger signal to generate in response to a first download of the remote one or more web pages.

4. The method of claim 1, wherein analyzing the retrieved content on display to determine information includes determining a classification of the object in the image of the content on display.

5. The method of claim 4, wherein the classification correlates the object to one or more classes of merchandise items.

6. The method of claim 1, wherein analyzing the retrieved content on display to determine information includes determining image data for the object in the image of the content on display.

7. The method of claim 6, wherein the image data corresponds to a signature of the object or of a portion of the object.

8. The method of claim 6, wherein analyzing the retrieved content on display to determine information includes identifying one or more visual characteristics of the object.

9. The method of claim 8, wherein the one or more visual characteristics correspond to one or more of a color, a pattern, or a shape of either the object or a portion of the object.

10. The method of claim 1, wherein using the determined information includes, responsive to detecting the selection, generating a search criteria from the image data that is associated with the object in the image of the content on display.

11. The method of claim 10, wherein using the determined information includes performing a search using the search criteria, wherein the search criteria is used to identify one or more records that each contain images of objects that satisfy the search criteria.

12. The method of claim 11, wherein performing a search using the search criteria includes using an index.

13. A system for analyzing images, the system comprising:

a retrieval component that is triggerable by activity at a remote web page to access the web page and to retrieve content on display at the remote web page, wherein the content on display includes one or more images;

one or more modules that are configured to identify one or more objects in the one or more images, and to determine whether the one or more objects are of a class that are designated as being of interest.

14. The system of claim 13, wherein the one or more modules are configured to identify one or more objects by analyzing each image of the content on display to determine whether any object designated as being of a class of interest is identifiable from the image.

15. The system of claim 13, wherein the one or more modules are configured to identify one or more objects by analyzing the one or more images to segment a given object in the image that is designated as being of a class of interest from another portion of the image.

16. The system of claim 15, wherein the one or more modules are configured to identify one or more objects by analyzing the one or more images to determine image data that represents the given object in the image.

17. The system of claim 16, wherein the image data corresponds to one or more signatures, wherein each signature represents one or more of (i) the given object as a whole, (ii) a portion of the object, or (iii) a visual characteristic of the given object.

18. The system of claim 16, wherein the one or more modules are configured to activate the one or more images on display to enable a user of the one or more images to select the given object from the image and to trigger use of the image data by a system that is remote to the web page on which the content is on display.

19. The system of claim 18, wherein the system includes a visual search engine, and wherein the image data is used to identify one or more records, from a plurality of records stored with the visual search engine, that include images of objects that satisfy a criteria of similarity with the image of the given object.

20. The system of claim 13, wherein the visual search engine generates a result that includes a panel for each identified record.

21. The system of claim 20, wherein each panel in the result displays a merchandise item for sale, and provides a link to a network location where the merchandise item is purchasable.

22. The system of claim 13, further comprising a triggering component that resides with the one or more web pages and triggers the retrieval component when the content on display is first downloaded.

23. The system of claim 13, wherein the one or more web pages include a blog or a webpage that includes images displayed in free form.

24. The system of claim 15, wherein the given object of the class of interest corresponds to one of apparel, clothing or jewelry.

25. The system of claim 13, wherein the one or more modules are configured to identify one or more objects from a first image by (i) detecting one or more markers for a first class of objects, and (ii) locating an object of the first class from the one or more markers.

26. The system of claim 25, wherein the one or more modules are configured to detect at least a first marker that is physically spaced apart in the first image from the object of the first class.

27. The system of claim 25, wherein the one or more markers correspond to a marker that is indicative of a presence of a face in the first image.

28. The system of claim 27, wherein the one or markers include a marker corresponding to one of an eyes, an eye brow, a lip or mouth edge, a nose, or a nose tip.

29. The system of claim 27, wherein the one or more objects correspond to a clothing, an apparel, or a jewelry of a person of the face.

30. A method for analyzing images, the method comprising:

accessing a remote web page to retrieve an image on display at the remote web pages;

analyzing the image on display to detect a person shown in the image;

detecting one or more of a clothing, apparel, or jewelry worn by the person shown in the image; and

enabling at least a portion of the image corresponding to the detected one or more of the clothing, apparel or jewelry, to be selectable by a person viewing the remote web page.

31. The method of claim 30, further comprising:

responsive to the person viewing the remote web page selecting the portion of the image, displaying, or causing display, of information about the one or more of the clothing, apparel or jewelry.

32. The method of claim 31, wherein displaying, or causing display, of information includes performing a visual search using criteria that is determined or identified from the one or more of the clothing, apparel or jewelry.

33. The method of claim 32, wherein performing a visual search using criteria includes identifying or determining a signature as the criteria, wherein the criteria represents an appearance of the one or more of the clothing, apparel or jewelry.

34. The method of claim 30, wherein analyzing the image on display to detect a person shown in the image includes detecting a marker for a face of the person, and wherein detecting one or more of a clothing, apparel, or jewelry worn by the person shown in the image includes processing the image using the marker of the face as a reference point.

35. The method of claim 34, wherein processing the image using the marker of the face as a reference point includes determining a skin color of the face, and detecting a clothing from a variation in color in the image at a region of the image where the clothing is likely to be located in relation to the reference.

36. The method of claim 30, further comprising performing recognition of the detected person to determine an identity of the person.

37. The method of claim 30, wherein at least one of analyzing the image, and detecting one or more of a clothing, apparel, or jewelry is performed automatically and in response to a page view of the remote web page by a third-person.

38. The method of claim 30, wherein analyzing the image on display to detect a person includes performing one or more of a face detection or segmentation of a portion of the image that contains at least a part of the person.