LINK DISCOVERY FROM WEB SCRIPTS

• US 20090125469A1
• Filed: 11/09/2007
• Published: 05/14/2009
• Est. Priority Date: 11/09/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: QIAN, RICHARD J., MCDONALD, KIERAN RICHARD, AALETI, SRINATH REDDY
• Granted Patent: US 8,572,065 B2
• Time in Patent Office: Days
• US Class Current: 706/47
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A computer-implemented method, a computer system, and computer media for discovering links in scripts are provided. The computer system includes a crawler, a rules engine, and an index that are utilized to store links generated by scripts located in webpages in the index. The crawler traverses a network to locate webpages having scripts. The rules engine parses the located webpages and extracts the scripts based on rules that are satisfied by segments of the extracted scripts. The rules engine evaluates the segments of the extracted scripts to generate links. After the rules engine validates the links, the rules engine transmits the links to the index for storage.

Claims

1. A computer-implemented method for discovering links in a script, the method comprising:

receiving webpages associated with one or more scripts;

processing the webpages to locate the one or more scripts;

accessing rules corresponding to the one or more scripts;

parsing the one or more scripts based on the rules corresponding to the one or more scripts;

identifying segments of the one or more scripts that satisfy the rules; and

evaluating the identified segments of the one or more scripts to generate links.

2. The computer-implemented method of claim 1, wherein the links include uniform resource locators.

3. The computer-implemented method of claim 2, wherein the uniform resource locators reference webpages, videos, audio, and multimedia content.

4. The computer-implemented method of claim 1, wherein the one or more scripts comprise external scripts, inline scripts, and event handler scripts.

5. The computer-implemented method of claim 1, wherein the one or more scripts are Javascripts.

6. The computer-implemented method of claim 5, wherein processing the webpages to locate the Javascripts comprises:

extracting markup language tags from the webpages to locate function calls, variables, and constants; and

identifying script elements and non-script elements based on the markup language tags.

7. The computer-implemented method of claim 6, wherein rules are applied to the extracted function calls, variables and constants to generate the links.

8. One or more computer-readable media having computer-executable instructions embodied thereon that perform a method for generating an index that stores links discovered in scripts, the method comprising:

crawling a network to locate webpages;

storing in an index metadata corresponding to each located webpage;

parsing the located webpages to identify scripts associated with the located webpages;

retrieving rules that check the identified scripts for variables, functions, or events that generate links;

evaluating the variables, functions, or events to verify the validity of the generated links; and

adding the generated links to the index when the generated links are verified.

9. The media of claim 8, wherein the link is a uniform resource locator.

10. The media of claim 8, wherein the scripts are generated by external files, inline code, or event handlers.

11. The media of claim 8, wherein the links are generated from data received from XML files processed by the identified scripts.

12. The media of claim 11, wherein the identified scripts are Javascripts.

13. The media of claim 8, wherein additional rules are generated for derived functions, in the identified scripts, that encapsulate a function associated with the retrieved rules.

14. The computer system of claim 13, wherein the derived functions generate links.

15. The computer system of claim 13, wherein the additional rules have a scope that is limited to a subset of the located webpages or a segment of the identified scripts in the located webpages on the network.

16. A computing system for discovering links in a script of a webpage, the system comprising:

a crawler accessing web pages to identify scripts associated with the webpages, wherein the webpages are HTML pages;

a rules engine that parses the identified scripts and evaluates portions of the identified script based on rules that detect link-generating segments of the identified scripts, wherein the segments of the identified scripts are evaluated based on variables and expressions specified in the identified scripts and matching function patterns located in the segment and the rules to detect links generated by the identified scripts; and

an index to store the detected links and metadata for the detected links.

17. The computing system of claim 16, wherein the metadata comprises types of content associated with detected links, types of files associated with the detected links, and display sizes of the content associated with the detected links.

18. The computing system of claim 16, wherein the rules engine verifies the detected links.

19. The computing system of claim 16, wherein the identified scripts correspond to a multimedia player that utilizes client-side scripts associated with XML files, and the rules engine parses the XML files to generate data and performs string transformations on the generated data to extract links from the XML files.

20. The computing system of claim 16, wherein the rules check functions, variables, and event handlers specified in the identified scripts and a subset of the rules are scoped to correspond to a subset of the webpages.