System and method for preventing web crawler access

• US 20100023751A1
• Filed: 07/15/2009
• Published: 01/28/2010
• Est. Priority Date: 07/24/2008
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: He, Ling
• Granted Patent: US 8,762,705 B2
• Time in Patent Office: Days
• US Class Current: 713/150
• CPC Class Codes: H04L 2463/041 using an encryption or decryption engine integrated in transmitted data
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/0457 wherein the sending and receiving network entities apply dynamic encryption, e.g. stream encryption cryptographic mechanisms or cryptographic arrangements for stream encryption H04L9/065
• CPC Class Codes: H04L 63/1466 Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks

Abstract

Preventing web crawler access includes receiving a request for a webpage that includes web content that should be protected from a web crawler, encrypting the web content to be protected to generate encrypted content and responding to the request, including sending the encrypted content and a decryption instruction. The decryption instruction is configured to allow a web browser to decrypt the encrypted content.

Claims

1. A method for preventing web crawler access, comprising:

receiving a request for a webpage that includes web content that should be protected from a web crawler;

encrypting the web content to be protected to generate encrypted content; and

responding to the request, including sending the encrypted content and a decryption instruction, wherein the decryption instruction is configured to allow a web browser to decrypt the encrypted content.

2. The method of claim 1, wherein encrypting the web content includes:

obtaining a mapping table of two character sequences, the first character sequence comprising a plurality of characters and the second character sequence comprising the plurality of characters in a random order; and

replacing each character in the web content to be protected with a corresponding character in the mapping table.

3. The method of claim 1, wherein encrypting the web content includes:

obtaining a mapping table of two character sequences, the first character sequence comprising a plurality of characters and the second character sequence comprising the plurality of characters in a random order;

converting the web content to be protected from a first encoding scheme to a second encoding scheme;

replacing each character in the web content to be protected with a corresponding character in the mapping table to generate encrypted web content; and

marking the encrypted web content using at least one Hypertext Mark-up Language (HTML) element tag.

4. The method of claim 3, wherein the first encoding scheme is American Standard Code for Information Interchange (ASCII) and the second encoding scheme is Unicode.

5. The method of claim 1, wherein the decryption instruction includes a script.

6. The method of claim 1, wherein the decryption instruction includes JavaScript code.

7. The method of claim 1, wherein the decryption instruction is inserted into a random location on the webpage.

8. A system comprising:

an input interface configured to receive a request for a webpage that includes web content that should be protected from a web crawler; and

a processor coupled to the input interface, configured to;

encrypt the web content to be protected to generate encrypted content; and

respond to the request, including send the encrypted content and a decryption instruction, wherein the decryption instruction is configured to allow a web browser to decrypt the encrypted content.

9. The system of claim 8, wherein encrypting the web content includes:

obtaining a mapping table of two character sequences, the first character sequence comprising a plurality of characters and the second character sequence comprising the plurality of characters in a random order; and

replacing each character in the web content to be protected with a corresponding character in the mapping table.

10. The system of claim 8, wherein encrypting the web content includes:

obtaining a mapping table of two character sequences, the first character sequence comprising a plurality of characters and the second character sequence comprising the plurality of characters in a random order;

converting the web content to be protected from a first encoding scheme to a second encoding scheme;

replacing each character in the web content to be protected with a corresponding character in the mapping table to generate encrypted web content; and

marking the encrypted web content using at least one Hypertext Mark-up Language (HTML) element tag.

11. The system of claim 10, wherein the first encoding scheme is American Standard Code for Information Interchange (ASCII) and the second encoding scheme is Unicode.

12. The system of claim 8, wherein the decryption instruction includes a script.

13. The system of claim 8, wherein the decryption instruction includes JavaScript code.

14. The system of claim 8, wherein the decryption instruction is inserted into a random location on the webpage.

15. A computer program product for preventing web crawler access, the computer program product being embodied in a computer readable storage medium and comprising computer instructions for:

receiving a request for a webpage that includes web content that should be protected from a web crawler;

encrypting the web content to be protected to generate encrypted content; and

responding to the request, including by sending the encrypted content and a decryption instruction, wherein the decryption instruction is configured to allow a web browser to decrypt the encrypted content.