Interactive web crawler

• US 8,538,949 B2
• Filed: 06/17/2011
• Issued: 09/17/2013
• Est. Priority Date: 06/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Chao, Wang, Yi-Min, Zhaou, Chao
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US13/163,001
• Publication Number:
  US 20120323881A1
• Time in Patent Office: 823 Days
• Field of Search: 707/709, 707/999.107, 715/224, 709/203, 709/219
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

The claimed subject matter provides a system or method for web crawling hidden files. An exemplary method includes loading a web page with a browser agent, and executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values. A list of form controls may be retrieved from the web page using the browser agent, and the controls may be analyzed using a driver component. Form control values may be sent from the driver component to the browser agent, and an event may be submitted to the web page by the browser agent or scripted content may be run to trigger operations on the web page corresponding to the form control values. A URL may be generated for various form control values using a generalizer.

Claims

1. A method of web crawling hidden files, comprising:

loading a web page with a browser agent;

executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values;

retrieving a list of form controls from the web page using the browser agent;

analyzing the form controls using a driver component of a crawler;

sending form control values from the driver component to the browser agent;

submitting an event to the web page by the browser agent or running any scripted content to trigger operations on the web page corresponding to the form control values;

generating a URL for various form control values using a generalizer; and

re-fetching, using the browser agent, web page content, a new list of form controls, and corresponding values for a new control that is dependent upon one of the form controls, wherein the browser agent re-fetches until all form controls are executed.

2. The method recited in claim 1, comprising re-fetching new form control lists and corresponding values for any dependent controls or web page content using the browser agent.

3. The method recited in claim 1, wherein the driver component uses a knowledge base that provides knowledge relevant to the driver component'"'"'s decision making.

4. The method recited in claim 1, wherein the driver component incorporates real world constraints into decision making.

5. The method recited in claim 1, comprising a utility estimator to estimate the potential gain of each form control value setting under real world constraints.

6. The method recited in claim 1, wherein the generalizer provides a seed URL for a crawler, or the generalizer generates another valid URL based on generalizations or incremental indexing.

7. The method recited in claim 1, comprising iteratively generating another URL until all controls and their dependent controls have been analyzed, until a certain number of URLs have been obtained, or until all resources have been used.

8. A system for web crawling hidden files, the system comprising:

a processing unit; and

a system memory, wherein the system memory comprises code configured to direct the processing unit to;

load a web page via a browser agent module;

execute any dynamic elements hosted on the web page using the browser agent module to insert pre-determined values;

retrieve a list of form controls from the web page using the browser agent module;

analyze the form controls using a driver component module of a crawler;

send form control values from the driver component module to the browser agent module;

generate a URL for various form control values using a generalize module; and

re-fetch, using the browser agent, web page content, a new list of form controls, and corresponding values for a new control that is dependent upon one of the form controls, wherein the browser agent re-fetches until all form controls are executed.

9. The system recited in claim 8, comprising re-fetching new form control lists and corresponding values for any dependent controls or web page content using the browser agent module.

10. The system recited in claim 8, wherein the driver component module uses a knowledge base that provides knowledge relevant to the driver component module'"'"'s decision making.

11. The system recited in claim 8, wherein the driver component module incorporates real world constraints into decision making.

12. The system recited in claim 8, comprising a utility estimator to estimate the potential gain of each form control value setting under real world constraints.

13. The system recited in claim 8, generalizer module provides a seed URL for a crawler, or the generalize module generates another valid URL based on generalizations or incremental indexing.

14. The system recited in claim 8, comprising iteratively generating another URL until all controls and their dependent controls have been analyzed, until a certain number of URLs have been obtained, or until all resources have been used.

15. One or more computer-readable storage media, comprising code configured to direct a processing unit to:

load a web page with a browser agent;

execute any forms hosted on the web page using the browser agent to insert pre-determined values;

retrieve a list of form controls from the web page using the browser agent;

analyze the form controls using a driver component of a crawler;

send form control values from the driver component to the browser agent;

generate a URL for various form control values using a generalizer; and

re-fetch, using the browser agent, web page content, a new list of form controls, and corresponding values for a new control that is dependent upon one of the form controls, wherein the browser agent re-fetches until all form controls are executed.

16. The one or more computer-readable storage media recited in claim 15, wherein the driver component uses a knowledge base that provides knowledge relevant to the driver component'"'"'s decision making, or the driver component incorporates real world constraints into decision making.

17. The one or more computer-readable storage media recited in claim 15, comprising a utility estimator to estimate the potential gain of each form control value setting under real world constraints.

18. The one or more computer-readable storage media recited in claim 15, the generalizer provides a seed URL for a crawler or the generalizer generates another valid URL based on generalizations or incremental indexing.

19. The one or more computer-readable storage media recited in claim 15, comprising iteratively generating another URL until all controls and their dependent controls have been analyzed, until a certain number of URLs have been obtained, or until all resources have been used.