Interactive web crawler
First Claim
1. A method of web crawling hidden files, comprising:
- retrieving a list of form controls from a web page;
retrieving one or more candidate values for one of the form controls;
retrieving an additional form control candidate value for one of the form controls, the additional form control candidate value not being shown in a static HTML description of the web page;
generating form control values for the form controls based on the candidate values, the additional form control candidate value, and a knowledge base comprising characterizations of possible form control values, and N-grams generated from search query logs;
submitting an event to the web page using the form control values;
generating a URL for the form control values to crawl for the hidden files; and
generating a plurality of URLs, the plurality of URLs being generated based on one or more constraints, the form control values being generated such that an estimated utility of the form control values is increased, the estimated utility being based on compliance with the constraints.
1 Assignment
0 Petitions
Accused Products
Abstract
The claimed subject matter provides a system or method for web crawling hidden files. An example method includes loading a web page with a browser agent, and executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values. A list of form controls may be retrieved from the web page using the browser agent, and the controls may be analyzed using a driver component. Form control values may be sent from the driver component to the browser agent, and an event may be submitted to the web page by the browser agent or scripted content may be run to trigger operations on the web page corresponding to the form control values. A URL may be generated for various form control values using a generalizer.
26 Citations
15 Claims
-
1. A method of web crawling hidden files, comprising:
-
retrieving a list of form controls from a web page; retrieving one or more candidate values for one of the form controls; retrieving an additional form control candidate value for one of the form controls, the additional form control candidate value not being shown in a static HTML description of the web page; generating form control values for the form controls based on the candidate values, the additional form control candidate value, and a knowledge base comprising characterizations of possible form control values, and N-grams generated from search query logs; submitting an event to the web page using the form control values; generating a URL for the form control values to crawl for the hidden files; and generating a plurality of URLs, the plurality of URLs being generated based on one or more constraints, the form control values being generated such that an estimated utility of the form control values is increased, the estimated utility being based on compliance with the constraints. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A system for web crawling hidden files, the system comprising:
-
a processing unit; and a system memory, wherein the system memory comprises code configured to direct the processing unit to; retrieve a list of form controls from a web page; retrieve one or more candidate values for one of the form controls; retrieve an additional form control candidate value for one of the form controls, the additional form control candidate value not being shown in a static HTML description of the web page; generate form control values for the form controls based on the candidate values, the additional form control candidate value, and a knowledge base comprising characterizations of possible form control values, and N-grams generated from search query logs; submit an event to the web page using the form control values; generate a URL for the form control values to crawl for the hidden files; and generate a plurality of URLs, the plurality of URLs being generated based on one or more constraints, the form control values being generated such that an estimated utility of the form control values is increased, the estimated utility being based on compliance with the constraints. - View Dependent Claims (8, 9, 10, 11, 12, 13, 14, 15)
-
Specification