Method and apparatus for an application crawler

• US 7,584,194 B2
• Filed: 11/22/2005
• Issued: 09/01/2009
• Est. Priority Date: 11/22/2004
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Truveo Incorporated (Warner Bros. Discovery, Inc.)
• Inventors: Beguelin, Adam L, Tuttle, Timothy D., Kocks, Peter F.
• Primary Examiner(s): Ali; Mohammad
• Assistant Examiner(s): Alvesteffer; Jason L
• Application Number:
  US11/286,269
• Publication Number:
  US 20060230011A1
• Time in Patent Office: 1,379 Days
• Field of Search: 707/3, 707/4, 707/5, 707/6, 707/101, 707/102, 707/104.1, 707/10, 705/210, 705/234
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A computer-implemented method is provided for searching for files on the Internet. In one embodiment, the method may provide an application crawler that assembles and dynamically instantiates all components of a web page. The instantiated web application may then be analyzed to locate desired components on the web page. This may involve finding and analyzing all clickable items in the application, driving the web application by injecting events, and extracting information from the application and writing it to a file or database.

Claims

1. A computer-implemented method for searching for files on the Internet, the method comprising:

finding a target URL;

downloading an HTML file for the target URL;

downloading supplementary data files used to build a web application, based on information in the HTML file;

assembling application components from the supplementary data files and the HTML file;

instantiating application components to create the web application;

applying data-query interfaces to all media objects in the web application that may contain useful data;

loading a pre-defined Application template or generating and automatically defining an Application template;

applying the Application template to extract all of the desired information from the web application;

saving the desired information to a file or database as a structured data information record;

examining all components in the web application to identify all possible components that could respond to a mouse event or form a clickable item;

determining which clickable items have appeared since a last simulated mouse event;

storing new clickable items in an appropriate data structure on a storage medium, wherein the appropriate data structure is a new branch of a clickable item tree containing all clickable items in the application at all possible application states;

simulating a mouse click on a first clickable item in a current branch of the clickable item tree; and

repeating this method until the entire clickable item tree has been traversed including, while continuing to instantiate the application components to create the web application, at a subsequent point in time, relating information gathered from examining subsequently loaded and instantiated components of the web application that are displayed at the subsequent point in time.

2. The method of claim 1 wherein the instantiating step includes at least one of the following:

rendering HTML and constructing a Document Object Model;

applying style sheets;

executing scripts in an appropriate script interpreter;

activating any controls or plug-ins;

launching video or audio streams;

launching animations;

or

executing HTML behavior scripts.

3. A computer program product comprising:

a computer usable storage medium and computer readable code embodied on said computer usable storage medium, the computer readable code comprising computer executable instructions that, as executed by a processor, cause a computer implemented system to perform a method for;

finding a target URL;

downloading an HTML file for the target URL;

downloading supplementary data files used to build a web application, based on information in the HTML file;

assembling application components from the supplementary data files and the HTML file;

instantiating application components to create the web application;

applying data-query interfaces to all media objects in the web application that may contain useful data;

loading a pre-defined Application template or generating and automatically defining an Application template;

applying the Application template to extract all of the desired information from the web application;

saving the desired information to a file or database as a structured data information record;

examining all components in the web application to identify all possible components that could respond to a mouse event or form a clickable item;

determining which clickable items have appeared since a last simulated mouse event;

storing new clickable items in an appropriate data structure on a storage medium, wherein the appropriate data structure is a new branch of a clickable item tree containing all clickable items in the application at all possible application states;

simulating a mouse click on a first clickable item in a current branch of the clickable item tree; and

repeating this method until the entire clickable item tree has been traversed including, while continuing to instantiate the application components to create the web application, at a subsequent point in time, relating information gathered from examining subsequently loaded and instantiated components of the web application that are displayed at the subsequent point in time.

4. The computer program product recited in claim 3, wherein the computer executable instructions that, as executed by a processor, cause a computer implemented system to perform instantiating application components to create a web application, further performs at least one of:

rendering HTML and constructing a Document Object Model;

applying style sheets;

executing scripts in an appropriate script interpreter;

activating any controls or plug-ins;

launching video or audio streams;

launching animations;

or

executing HTML behavior scripts.

5. A computer system having a storage medium having computer-executable code stored thereon and a processor, the computer system comprising:

an application crawler having programming code configured to, as executed by a processor;

find a target URL;

download an HTML file for the target URL;

download supplementary data files used to build a web application, based on information in the HTML file;

assemble application components from the supplementary data files and the HTML file;

instantiate application components to create the web application;

apply data-query interfaces to all media objects in the web application that may contain useful data;

load a pre-defined Application template or generating and automatically defining an Application template;

apply the Application template to extract all of the desired information from the web application;

save the desired information to a file or database as a structured data information record;

examine all components in the web application to identify all possible components that could respond to a mouse event or form a clickable item;

determine which clickable items have appeared since a last simulated mouse event;

store new clickable items in an appropriate data structure on a storage medium, wherein the appropriate data structure is a new branch of a clickable item tree containing all clickable items in the application at all possible application states;

simulate a mouse click on a first clickable item in a current branch of the clickable item tree; and

repeat this method until the entire clickable item tree has been traversed including, while continuing to instantiate the application components to create the web application, at a subsequent point in time, relating information gathered from examining subsequently loaded and instantiated components of the web application that are displayed at the subsequent point in time.

6. The system of claim 5 wherein the programming code of the application crawler is further configured to, when instantiating application components to create the web application, perform at least one of the following:

render HTML and constructing a Document Object Model;

apply style sheets;

execute scripts in an appropriate script interpreter;

activate any controls or plug-ins;

launch video or audio streams;

launch animations;

or

execute HTML behavior scripts.