System and method for preventing automated crawler access to web-based data sources using a dynamic data transcoding scheme

• US 6,938,170 B1
• Filed: 07/17/2000
• Issued: 08/30/2005
• Est. Priority Date: 07/17/2000
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Myllymaki, Jussi Petri, Kraft, Reiner, Ruvolo, Joann
• Primary Examiner(s): Morse, Gregory
• Assistant Examiner(s): HENEGHAN, MATTHEW E
• Application Number:
  US09/617,460
• Time in Patent Office: 1,870 Days
• Field of Search: 707/5, 707/9, 707/101, 709/203, 709/236, 713/189, 713/190, 713/193, 713/194, 713/200, 713/201, 715/523
• US Class Current: 726/23
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 21/50 Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A protection system and associated method prevent the automatic crawler access to a company'"'"'s web-based data, without impacting the ability of an interactive user, such as a consumer, to access the data and to conduct regular business transactions. In one embodiment, the protection system will not prevent the crawler from downloading data; rather, the data will be rendered non-extractable by the crawler. In another embodiment, the protection system will prevent crawler access to the data. To this end, the protection system uses any one or a combination of the following six transcoding techniques: Transcoding technique that changes the web page structure; transcoding technique that changes the web page content; transcoding technique that selectively changes web page variable names; transcoding technique that selectively converts text to images in the web page; transcoding technique that alters form values when executed; and/or transcoding technique that generates a substantial portion of, or the entire web page when executed.

Claims

1. A system for preventing automated crawler access to data from a network-based data source, comprising:

a transcoding proxy for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source;

wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form structure;

wherein the transcoding technique changes the form structure by shifting the position of the data in the form;

wherein the transcoding technique shifts the position of form data by inserting any one or more of;

a character, a string of characters, a space, non-textual data, a blank row, a blank column, and/or an empty table; and

wherein the transcoding technique shifts the position of form data by nesting the data in a table within a table.

2. The system according to claim 1,

wherein the transcoding proxy concurrently permits a browser to access the data source.

3. The system according to claim 1, wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form content.

4. The system according to claim 1, wherein the data retrieved from the data source include a variable with a name;

and

wherein the transcoding proxy utilizes a transcoding technique that dynamically changes the name of the variable.

5. The system according to claim 1, wherein the data retrieved from the data source include a textual content;

and

wherein the transcoding proxy utilizes a transcoding technique that dynamically changes the textual content into non-textual content.

6. The system according to claim 5, wherein the transcoding technique replaces the textual content by a corresponding image, in order to render selected terms difficult to be searched automatically by the crawler.

7. The system according to claim 1, wherein the data retrieved from the data source includes a form containing a variable with a value;

wherein the transcoding proxy utilizes a transcoding technique that incorporates an executable application in the form; and

wherein when the executable application is implemented, it dynamically alter the value of the variable.

8. The system according to claim 7, wherein the variable includes a label associated with the variable;

wherein the transcoding technique further transforms the label into a synthetic label; and

wherein the executable application recalculates the variable value in order to prevent a server from authenticating the crawler access to the data source.

9. The system according to claim 1, wherein the data retrieved from the data source is collected in a page;

wherein the transcoding proxy utilizes a transcoding technique that incorporates an executable application in the page; and

wherein when the executable application is implemented, it dynamically generates the page and renders the data.

10. The system according to claim 9, wherein the transcoding technique renders the data using an HTML code.

11. A system for preventing automated crawler access to data from a network-based data source, comprising:

a transcoding proxy for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source;

wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form content; and

wherein the transcoding technique changes the form content by adding one or more content inserts that are substantially imperceptible to a browser user, in order to render selected terms in the page content difficult to be searched automatically by the crawler.

12. The system according to claim 11, wherein the content inserts include any one or more of:

a characters, a string of characters, non-textual data, and/or a string of images.

13. The system according to claim 12, wherein the transcoding technique calculates a synthetic name from a variable name, and replaces the variable name with the synthetic name, in order to render selected terms difficult to be searched automatically by the crawler.

14. The system according to claim 13, wherein the variable is contained in a data entry form;

wherein the data entry form further includes a session ID; and

wherein the transcoding technique recalculates the variable name from the synthetic name to enable the data to be rendered by the browser.

15. A method for preventing automated crawler access to data from a network-based data source, comprising the steps of:

automatically permutating data retrieved from the data source, to prevent a crawler from automatically accessing the data source by rendering the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source; and

utilizing a transcoding technique that dynamically changes a form structure;

wherein the transcoding technique changes the form structure by shifting the position of the data in the form; and

wherein the transcoding technique shifts the position of form data by nesting the data in a table within a fable.

16. The method according to claim 15, wherein the data contain any one or more of:

a variable with a name, a textual content, or a value;

wherein the data are collected in one or more of;

a form or a page; and

wherein automatically permuting data and preventing crawler access include using one or more of the following transcoding techniques;

a first transcoding technique that dynamically changes a form structure;

a second transcoding technique that dynamically changes a form content;

a third transcoding technique that dynamically changes the name of the variable;

a fourth transcoding technique that dynamically changes the textual content into a non-textual content;

a fifth transcoding technique that incorporates an executable application in the form, and wherein when the executable application is implemented, it dynamically alter the value of the for variable; and

/or

a sixth transcoding technique that incorporates an executable application in the page, and wherein when the executable application is implemented, it dynamically generates the page and renders the data.

17. A system for preventing automated crawler access to data from a network-based data source, comprising:

a transcoding proxy for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source;

wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form structure;

wherein the transcoding technique changes the form structure by shifting the position of the data in the form; and

wherein the transcoding technique shifts the position of form data by nesting the data in a table within a table.

18. A computer program product including a plurality of executable instruction codes stored on a computer readable medium, for preventing automated crawler access to data from a network-based data source, comprising:

a first set of instruction codes for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source;

a second set of instruction codes for dynamically changing a form structure;

wherein the second set of instruction codes changes the form structure by shifting the position of the data in the form; and

wherein the second set of instruction codes shifts the position of form data by nesting the data in a table within a table.