Methods for analyzing dynamic web pages

• US 9,405,833 B2
• Filed: 09/14/2012
• Issued: 08/02/2016
• Est. Priority Date: 11/22/2004
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Tuttle, Timothy D., Beguelin, Adam L., Kocks, Peter F.
• Primary Examiner(s): Willoughby, Alicia
• Application Number:
  US13/618,968
• Publication Number:
  US 20130066848A1
• Time in Patent Office: 1,418 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

A computer-implemented method is provided for searching for files on the Internet. In one embodiment, the method may provide an application crawler that assembles and dynamically instantiates all components of a web page. The instantiated web application may then be analyzed to locate desired components on the web page. This may involve finding and analyzing all clickable items in the application, driving the web application by injecting events, and extracting information from the application and writing it to a file or database.

Claims

1. A method comprising:

loading a web page;

identifying an extractor template for extracting information from the web page, wherein the extractor template comprises timing instructions for extracting information from the web page;

identifying a first object loaded in a web page;

simulating user input that manipulates the identified first object;

identifying, by at least one processor, a video loaded in response to the simulated user input;

extracting first information from the video;

in accordance with the timing instructions of the extractor template, skipping a commercial associated with the video;

continuing, in accordance with the timing instructions of the extractor template, to extract information from the video by extracting second information from the video after skipping the commercial associated with the video; and

aggregating the first information from the video and the second information from the video in an index.

2. The method of claim 1, further comprising indexing an object model of the web page.

3. The method of claim 2, further comprising:

traversing a tree of the object model to identify a second object that is user manipulable and loaded in the web page;

applying a data query interface to the second object;

simulating user input that manipulates the second object and causes the second object to change state; and

extracting information from the second object.

4. The method of claim 3, wherein the identified first object is a media player.

5. The method of claim 1, further comprising post-processing the information extracted from the identified video.

6. The method of claim 5, further comprising applying voice-to-text recognition to at least a portion of the information extracted from the video.

7. The method of claim 1, further comprising reexamining the web page in accordance with at least one of an automated algorithm or the extractor template.

8. The method of claim 1, further comprising identifying a second object loaded in the web page based on a spatial proximity of the second object with respect to one or more other objects on the web page.

9. The method of claim 1, further comprising identifying a second object loaded in the web page based on one or more visual characteristics of the second object.

10. The method of claim 1, further comprising identifying a second object loaded in the web page based on a time the second object is loaded in the web page.

11. The method of claim 1, further comprising:

extracting the first information using a first extraction technique comprising one of a Document Object Model implementation for a browser, a scripting engine, an XML parsing engine, a Cascading Style Sheet engine, a network I/O library, an HTML parsing and rendering engine, an engine for executing embedded controls, or an engine for rendering web applications; and

extracting the second information using a second extraction technique comprising one of an XSL engine, an XPATH implementation, a regular expression engine, a script execution engine, an embedded object inspector, a network transport proxy, a multimedia stream proxy, a software bridge, a taxonomy engine for categorizing metadata, or a text parsing and processing engine.

12. The method of claim 1, further comprising cascading a first extraction technique and a second extraction technique to further refine the information extracted from the video.

13. A non-transitory computer-readable medium including a set of instructions that, when executed, cause at least one processor to perform the steps comprising:

loading a web page;

identifying an extractor template for extracting information from the web page, wherein the extractor template comprises timing instructions for extracting information from the web page;

identifying a first object loaded in a web page;

simulating user input that manipulates the identified first object;

identifying a video loaded in response to the simulated user input;

extracting first information from the video;

in accordance with the timing instructions of the extractor template, skipping a commercial associated with the video;

continuing, in accordance with the timing instructions of the extractor template, to extract information from the video by extracting second information from the video after skipping the commercial associated with the video; and

aggregating the first information about the video and the second information about the video in an index.

14. The non-transitory computer-readable medium of claim 13, further comprising instructions that, when executed, cause at least one processor to perform the step of identifying, in accordance with an automated algorithm, a second object loaded in the web page.

15. The non-transitory computer-readable medium of claim 14, further comprising instructions that, when executed, cause at least one processor to perform the steps of:

loading the extractor template; and

identifying the second object based at least in part on the extractor template.

16. The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed, cause at least one processor to perform the step of identifying the second object based on a spatial proximity specified by the extractor template.

17. The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed, cause at least one processor to perform the step of identifying the second object based on one or more visual characteristics specified by the extractor template.

18. The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed, cause at least one processor to perform the step of identifying the second object based on a time specified by the extractor template.

19. A method comprising:

monitoring execution of a web page;

identifying an extractor template for extracting information from the web page, wherein the extractor template comprises timing instructions for extracting information from the web page;

applying, by at least one processor, the extractor template to the web page to identify a first object loaded in the web page;

simulating user input that manipulates the identified first object;

identifying, by the at least one processor, a video loaded in response to the simulated user input;

extracting first information from the video;

in accordance with the timing instructions of the extractor template, skipping a commercial associated with the video;

continuing, in accordance with the timing instructions of the extractor template, to extract information from the video after skipping the commercial by extracting second information from the video; and

aggregating the first information from the video and the second information from the video in an index.

20. The method of claim 19, further comprising detecting an advertisement provided in connection with the video.

21. The method of claim 19, further comprising monitoring changes in at least a portion of a screen buffer used to display the web page.

22. The method of claim 21, further comprising detecting events triggered by a media player embedded in the web page.

23. The method of claim 22, wherein the web page comprises one or more web applications.