METHODS AND APPARATUS TO AUTOMATICALLY CRAWL THE INTERNET USING IMAGE ANALYSIS

• US 20100080411A1
• Filed: 09/29/2008
• Published: 04/01/2010
• Est. Priority Date: 09/29/2008
• Status: Abandoned Application

• Current Assignee: Nielsen Company (US) LLC (RR Donnelley & Sons Company)
• Original Assignee: Nielsen Company (US) LLC (RR Donnelley & Sons Company)
• Inventors: Deliyannis, Alexandros
• Application Number:
  US12/240,756
• Publication Number:
  US 20100080411A1
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus to automatically crawl the Internet using image analysis are disclosed. An example method to visually identify components of a web page includes rendering a web page in a web browser to generate an image, and visually analyzing at least a portion of the image with a machine to detect a region containing a possible web page component. The example method further includes automatically determining a type of the detected web page component and storing the web page component type and a location of the portion of the web page.

Claims

1. A method to visually identify components of a web page, comprising:

rendering a web page to generate an image;

visually analyzing at least a portion of the image with a machine to detect a region containing a possible web page component;

automatically determining a type of the possible web page component; and

storing the web page component type and a location of the portion of the web page.

2. A method as defined in claim 1, further comprising accessing one or more hints, comprising location information corresponding to the portion of the image to be visually analyzed.

3. A method as defined in claim 2, wherein the hints are based on a previously-rendered web page.

4. A method as defined in claim 2, wherein the hints are based on a web page template.

5. A method as defined in claim 1, wherein the type comprises at least one of a button, media content, a media player control, a hyperlink, a text area, an advertisement, an image, or a Flash component.

6. A method as defined in claim 1, further comprising rendering a second web page to generate a second image in response to determining the type is a hyperlink.

7. A method as defined in claim 1, wherein automatically determining the type comprises performing image analysis of the region.

8. A method as defined in claim 7, wherein the image analysis comprises at least one of edge detection;

image correlation;

or initiating an action in the region of the web page, and monitoring to detect a response to the action.

9. A method as defined in claim 7, wherein the at least a portion is a Flash component.

10. A method as defined in claim 1, further comprising generating a web page template corresponding to the web page.

11. A method as defined in claim 10, wherein generating the web page template comprises storing the web page component type and the location in a table.

12. A method as defined in claim 1, further comprising identifying media content associated with the web page component.

13. An apparatus to identify components in a web page, comprising:

an image generator to render an image of a web page based on web page information;

an image analyzer to automatically analyze the image to detect a web page component in the image and to generate location information corresponding to a location of the web page component in the image; and

a component identifier to automatically determine a type of the web page component by analyzing the image.

14. An apparatus as defined in claim 13, further comprising a template reader to access a web page template.

15. An apparatus as defined in claim 14, further comprising a hint analyzer to identify one or more locations to be analyzed based on the web page template.

16. An apparatus as defined in claim 13, further comprising a storage device to store the type of the web page component and the location information.

17. An apparatus as defined in claim 16, wherein the component identifier is to store the type of the web page component and the location information in a web page template in the storage device.

18. An apparatus as defined in claim 17, wherein the component identifier is to determine a second component type in a second rendered web page based on location information in the template.

19. An apparatus as defined in claim 17, wherein the component identifier is to modify at least one of the type of the web page component or the location information based on the second component type.

20. An article of manufacture comprising instructions, which, upon execution, cause a machine to:

render a web page to generate an image;

visually analyzing at least a portion of the image to detect a region containing a possible web page component;

automatically determine a type of the detected web page component; and

store the web page component type and a location of the portion of the web page.

21. An article of manufacture as defined in claim 20, wherein the instructions further cause the machine to access hints, comprising location information corresponding to the portion of the image to be visually analyzed.

23. An article of manufacture as defined in claim 21, wherein the hints are based on a web page template.

26. An article of manufacture as defined in claim 20, wherein automatically determining the type comprises performing image analysis of the region.

27. An article of manufacture as defined in claim 26, wherein image analysis comprises at least one of edge detection;

image correlation;

or initiating an action in the portion of the web page corresponding to the region, and monitoring to determine a response to the action.

29. An article of manufacture as defined in claim 20, wherein the instructions further cause the machine to generate a web page template corresponding to the web page.

22. (canceled)

24. (canceled)

25. (canceled)

28. (canceled)

30-52. -52. (canceled)