Method and system for extending the performance of a web crawler

• US 20020103823A1
• Filed: 02/01/2001
• Published: 08/01/2002
• Est. Priority Date: 02/01/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jackson, Jared J., Myllymaki, Jussi P.
• Granted Patent: US 6,988,100 B2
• Time in Patent Office: Days
• US Class Current: 715/501.1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing

Abstract

A proxy engine (108), in communication with a web crawler (100), extends the performance of the web crawler (100) by modifying hyperlink requests and creating synthetic hyperlink requests from data received from a web site (104). The proxy engine (108) converts hyperlinks to a method used by a target web site. The proxy engine receives data from a web site (104) located across a network (102), and then determines whether additional data from the web site (104) is extractable. In response to determining that additional data from the web site (104) is extractable, the proxy engine (108) creates at least one synthetic hyperlink for extracting the data from the web site (104). The proxy engine (108) then combines the at least one synthetic hyperlink with the data received from the website (104) to create combined data and then sends the combined data to the crawler (100).

Claims

1. A method comprising the steps of:

receiving data from a web site located across a network;

determining whether additional data from the web site is extractable;

in response to determining that additional data from the web site is extractable, creating at least one synthetic hyperlink for extracting the data from the web site;

combining the at least one synthetic hyperlink with the data received from the website to create combined data; and

sending the combined data to a crawler.

2. The method of claim 1, wherein the step of creating at least one synthetic hyperlink comprises the step of:

augmenting a hyperlink request with at least one parameter value appropriate for a form used by the web site to create at least one synthetic hyperlink.

3. The method of claim 1, wherein the step of creating at least one synthetic hyperlink comprises the step of:

augmenting a hyperlink request with at least one parameter value determined by executing a script contained in the data from the web site to create at least one synthetic hyperlink.

4. The method of claim 1, wherein the step of creating at least one synthetic hyperlink comprises the step of:

augmenting a hyperlink request with at least one parameter value determined by a script filter analyzing a script contained in the data from the web site to create at least one synthetic hyperlink.

5. The method of claim 1, wherein the step of creating at least one synthetic hyperlink comprises the step of:

augmenting a hyperlink request with at least one parameter value to create a synthetic hyperlink, the synthetic hyperlink indicating that it must be converted to a different method for extracting data from the web site.

6. A method comprising the steps of:

receiving a synthetic hyperlink request;

converting the synthetic hyperlink request to a method indicated by the synthetic hyperlink request to create a converted hyperlink request; and

sending the converted hyperlink request to a web site.

7. The method of claim 6, wherein the step of converting the synthetic hyperlink request comprises the steps of:

determining whether the synthetic hyperlink request indicates the use of a POST method; and

in response to determining that the synthetic hyperlink request indicates the use of a POST method, converting the synthetic hyperlink from a GET method to a POST method.

8. A system comprising:

at least one communication device for receiving data from a web site and sending data to a crawler; and

at least one computer processing device, communicatively coupled to the at least one communication device, for

analyzing data received from the web site to determine whether additional data is extractable from the web site by a hyperlink request,

creating at least one synthetic hyperlink for extracting data from the web site, and

combining the at least one synthetic hyperlink with data received from the web site in response to the hyperlink request.

9. The system according to claim 8, wherein the at least one computer processing device comprises:

a transcoder for

determining at least one parameter value for a form required by the data from the web site;

augmenting the hyperlink request with at least one parameter value to create the at least one synthetic hyperlink; and

combining the at least one synthetic hyperlink with the data received from the web site in response to the hyperlink request.

10. The system according to claim 8, wherein the at least one computer processing device comprises:

a transcoder, communicatively coupled to the at least one communication device, for

augmenting the hyperlink request with at least one parameter value to create the at least one synthetic hyperlink for each of the at least one parameter value, and

combining the at least one synthetic hyperlink with the data received from the web site; and

a script engine, communicatively coupled to the transcoder, for determining the at least one parameter value by executing a script contained in the data from the web site.

11. The system according to claim 8, wherein the at least one computer processing device comprises:

a transcoder, communicatively coupled to the at least one communication device, for

augmenting the hyperlink request with at least one parameter value to create the at least one synthetic hyperlink for each of the at least one parameter value, and

combining the at least one synthetic hyperlink with the data received from the web site; and

a script filter, communicatively coupled to the transcoder, for determining the at least one parameter value by analyzing the script contained in the data from the web site.

12. The system according to claim 8, wherein the at least one computer processing device comprises:

a transcoder for augmenting the hyperlink request with at least one parameter value to create the at least one synthetic hyperlink indicating that the at least one synthetic hyperlink must be converted to a different method for extracting data from the web site.

14. The system of claim 13, wherein the method converter is for converting the synthetic hyperlink request from a GET method to a POST method.

16. The computer readable medium of claim 15, wherein the computer instructions for creating at least one synthetic hyperlink comprise instructions for augmenting a hyperlink request with at least one parameter value for a form used by the web site to create the at least one synthetic hyperlink.

17. The computer readable medium of claim 15, wherein the computer instructions for creating at least one synthetic hyperlink comprise instructions for augmenting a hyperlink request with at least one parameter value determined by executing a script contained in the data from the web site to create the at least one synthetic hyperlink.

18. The computer readable medium of claim 15, wherein the computer instructions for creating at least one synthetic hyperlink comprise instructions for augmenting a hyperlink request with at least one parameter value determined by a script filter by analyzing a script contained in the data from the web site to create the at least one synthetic hyperlink.

13. A system for extending the performance of a crawler comprising:

at least one communication device for receiving a synthetic hyperlink request and sending data to a web site across a network, in accordance with the synthetic hyperlink request; and

a method converter, communicatively coupled to the at least one communication device, for converting the synthetic hyperlink request to a method compatible with the web site.

15. A computer readable medium including computer instructions for a computing system, the computer instructions comprising instructions for:

receiving data from a web site located across a network;

determining whether additional data from the web site is extractable;

in response to determining that additional data from the web site is extractable, creating at least one synthetic hyperlink for extracting the data from the web site;

combining the at least one synthetic hyperlink with the data received from the web site to create combined data; and

sending the combined data to a crawler.

19. A computer readable medium including computer instruction for a computer system, the computer instructions comprising instructions for:

receiving a synthetic hyperlink request from a crawler;

converting the synthetic hyperlink request to a method indicated by the synthetic hyperlink request to create a converted hyperlink request; and

sending the converted hyperlink request to a web site.

20. A computer readable medium including computer instruction for a computer system, the computer instructions comprising instructions for:

receiving a synthetic hyperlink request for extracting data from a web site;

converting the synthetic hyperlink request to a method indicated by the synthetic hyperlink request; and

in response to determining that the synthetic hyperlink request indicates the use of a POST method, converting the synthetic hyperlink request from a GET method to a POST method.