System and method for enabling image recognition and searching of remote content on display

• US 8,732,025 B2
• Filed: 07/13/2007
• Issued: 05/20/2014
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gokturk, Salih Burak, Chiao, Dan, Phillips, Jacquie, Moran, Mark, Vanhoucke, Vincent, Khan, Azhar, Lin, Xiaofan, Shah, Munjal, Miller, Andrew, Dalal, Navneet, Vu, Diem
• Primary Examiner(s): Smith, Jeffrey A.
• Assistant Examiner(s): Civan, Ethan D
• Application Number:
  US11/777,894
• Publication Number:
  US 20080082426A1
• Time in Patent Office: 2,503 Days
• Field of Search: 705 26- 27, 705 261- 272
• US Class Current: 705/26.1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0625 Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06Q 30/08 Auctions

Abstract

Images are analyzed by programmatic mechanisms for assessing one or more remote web pages to retrieve content on display at remote web pages. The retrieved images may be analyzed to determine information about an object shown in a corresponding images of the content on display. At least a portion of the object shown in the corresponding image of the content on display may be made selectable and associated with the determined information. This determined information may subsequently be used, in for example, search applications.

Claims

1. A computer-implemented method performed by one or more processors and comprising:

analyzing, using the one or more processors, content of a web page that includes an image, wherein analyzing the content includes;

analyzing text or metadata of the web page to determine one or more assumptions about a location or shape of a first object in the image;

performing image analysis on the image based on the one or more assumptions to detect the first object in the image;

segmenting the first object from a remainder of the image; and

determining information about the first object, based on the image analysis, using one or more sources other than the content of the web page;

providing at least a portion of the first object depicted in the image to be a selectable feature of the web page, separate from the remainder of the image, at least the portion of the first object being associated with the determined information;

receiving data indicating a selection input corresponding to at least the portion of the first object; and

performing a search operation to determine a set of objects that are deemed to be visually similar to the first object based on the determined information.

2. The method of claim 1, further comprising receiving a trigger signal from a site of the web page, and then accessing the web page to retrieve the content for analyzing in response to receiving the trigger signal.

3. The method of claim 2, further comprising remotely causing the trigger signal to generate in response to a download of the web page.

4. The method of claim 1, wherein analyzing the content includes determining a classification of the first object.

5. The method of claim 4, wherein the classification correlates the first object to a person.

6. The method of claim 1, wherein analyzing the content includes detecting the first object from image data in the content of the web page.

7. The method of claim 6, wherein analyzing the content includes determining a signature of the first object from the image data.

8. The method of claim 6, wherein analyzing the content includes identifying one or more visual characteristics of the first object from the image data.

9. The method of claim 8, wherein the one or more visual characteristics correspond to one or more of a color, a pattern, or a shape of at least the portion of the first object.

10. The method of claim 1, wherein performing the search operation includes identifying a record that includes an image of a second object that is deemed visually similar to the first object.

11. The method of claim 10, wherein performing the search operation includes using an index.

12. A system comprising:

one or more processors;

memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to;

analyze content of a web page that includes an image, including;

analyzing text or metadata of the web page to determine one or more assumptions about a location or shape of a first object in the image,

performing image analysis on the image based on the one or more assumptions to detect the first object in the image,

segmenting a first object from a remainder of the image, and

determining information about the first object, based on image analysis, using one or more sources other than the content of the web page;

provide at least a portion of the first object depicted in the image to be a selectable feature of the web page, separate from the remainder of the image, at least the portion of the first object being associated with the determined information; and

receive data indicating a selection input corresponding to at least the portion of the first object;

perform a search operation to determine a set of objects that are deemed to be visually similar to the first object based on the determined information.

13. The system of claim 12, wherein the one or more processors are configured to analyze the image of the web page to determine whether the first object is of a predetermined class.

14. The system of claim 13, wherein the predetermined class of the first object corresponds to at least one of apparel, clothing or jewelry.

15. The system of claim 13, wherein the one or more processors are configured to segment the first object from the remainder of the image by detecting whether the first object as depicted in the image includes one or more markers for the predetermined class of objects.

16. The system of claim 12, wherein the one or more processors are configured to determine a signature for the first object based on image data that comprises the image.

17. The system of claim 16, wherein the signature identifies at least one of the first object as a whole, the portion of the first object, or a visual characteristic of the first object.

18. The system of claim 12, wherein the one or more processors are configured to provide the image on the web page in an interactive form, so that a user is able to provide a selection input corresponding to the first object in order to cause the one or more processors to perform the search operation.

19. The system of claim 12, wherein the one or more processors are configured to generate a search criterion for a search engine based on the determined information in response to a user providing the selection input.

20. The system of claim 19, wherein the one or more processors are configured to provide a search result from the search engine that includes one or more images.

21. The system of claim 20, wherein each of the one or more images of the search result depicts a merchandise item for sale, and is provided with a link to a network location where the merchandise item is purchasable.

22. The system of claim 12, wherein the one or more processors are configured to provide a triggering component and a retrieval component, wherein the triggering component resides at a location associated with the web page and triggers the retrieval component when the content is downloaded.

23. The system of claim 12, wherein the first object corresponds to a clothing, an apparel, or a jewelry worn by a person.

24. A computer-implemented method performed by one or more processors and comprising:

analyzing, using the one or more processors, content of a web page that includes an image, wherein analyzing the content includes analyzing text or metadata of the web page to determine one or more assumptions about a location and shape of a person, and one or more of a clothing, apparel, or jewelry worn by the person;

performing image analysis on the image based on the one or more assumptions to detect the person and at least one of the clothing, apparel, or jewelry in the image;

segmenting, from the image, the person, and the detected one or more of the clothing, apparel, or jewelry;

determining information about the detected one or more of the clothing, apparel, or jewelry, based on the image analysis, using one or more sources other than the content of the web page;

providing at least a portion of the image corresponding to the detected one or more of the clothing, apparel, or jewelry, to be a selectable feature on the web page, separate from the person and a remainder of the image;

receive data indicating a selection input from a user that corresponds to the one or more of the clothing, apparel or jewelry; and

perform a search operation to determine a set of objects that are deemed to be visually similar to the one or more of the clothing, apparel or jewelry that corresponds to the selection input.

25. The method of claim 24, wherein performing the search operation includes using a criterion that is determined from the one or more of the clothing, apparel or jewelry that corresponds to the selection input.

26. The method of claim 25, wherein performing the search operation includes performing the search operation based on a criterion that is determined from a signature of the one or more of the clothing, apparel or jewelry that corresponds to the selection input.

27. The method of claim 24, wherein analyzing the content includes detecting a marker for a face of the person depicted in the image, and wherein segmenting one or more of the clothing, apparel, or jewelry worn by the person depicted in the image includes processing the image using the marker of the face as a reference point.

28. The method of claim 27, wherein processing the image using the marker of the face as the reference point includes determining a skin color of the face, and detecting a clothing from a variation in color in the image at a region of the image where the clothing is likely to be located in relation to the reference point.

29. The method of claim 24, wherein analyzing the content is performed automatically and in response to a page view of the web page by the user.

30. The method of claim 24, wherein analyzing the content includes performing face detection on a portion of the image that contains at least a part of the person.