Detecting a network crawler

• US 10,182,046 B1
• Filed: 06/23/2015
• Issued: 01/15/2019
• Est. Priority Date: 06/23/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Canavor, Darren Ernest, Strand, William Alexander
• Primary Examiner(s): Le, Khoi
• Application Number:
  US14/747,939
• Time in Patent Office: 1,302 Days
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: H04L 2463/144 Detection or countermeasures against botnets
• CPC Class Codes: H04L 63/0815 providing single-sign-on or federations
• CPC Class Codes: H04L 63/083 using passwords cryptographic mechanisms or cryptographic arrangements for entity authentication using a predetermined code H04L9/3226
• CPC Class Codes: H04L 63/102 Entity profiles
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1491 using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

Techniques for detecting a network crawler may be described. In particular, a request for information may be received from a computing system. Based on this request, a network-based document may be provided to the computing system. The network-based document may include a portion of the information, code, and an identifier of another network-based document. The code may be configured to, upon execution, determine whether the other network-based document was accessed prior to providing the network-based document to the computing system. An indication whether the other network-based document was accessed may be received from the computing system. The indication may be received based on an execution of the code at the computing system. Based on the indication, the request for the information may be determined as being associated with a network crawler hosted on the computing system.

Claims

1. A computer-implemented method, comprising:

receiving, by a computer system associated with an electronic marketplace, a request for a first web page of a web site of the electronic marketplace, the request received from a computing device;

inserting, by the computer system in the first web page, a universal resource locator (URL) of a second web page of the web site and code, the second web page inaccessible to web crawlers based at least in part on a robots exclusion protocol of the web site identifying the second web page, the URL inserted in markup language of the first web page, the code comprising statements of a programmatic scripting language in accordance with an ECMAScript standard and configured to, upon execution of the code at the computing device;

determine a presence or absence of the URL in a browser history stored at the computing device and, if the URL is present in the browser history, one or more of;

a state or a style attribute of the URL from the browser history, and

determine, based at least in part on the presence or absence of the URL, whether the second web page was accessed by the computing device;

providing, by the computer system, the first web page to the computing device based at least in part on the request;

receiving, by the computer system from the computing device, an indication that the second web page was not accessed prior to providing the first web page to the computing device, the indication received based at least in part on a determination that the URL is absent from the browser history or on a determination of the state or style attribute of the URL if the URL is present in the browser history, the determination based at least in part on an execution of the code at the computing device; and

determining, based at least in part on the indication, that the request for the first web page is associated with a web crawler hosted on the computing device.

2. The computer-implemented method of claim 1, further comprising providing content associated with the first web page to the computing device based at least in part on a network path, wherein the network path is determined based at least in part on the request being associated with the web crawler.

3. The computer-implemented method of claim 1, wherein the browser history is associated with a browser hosted on the computing device and configured to present the first web page.

4. The computer-implemented method of claim 3, wherein the indication that the second web page was not accessed is based on one or more of:

the state or the style attribute of the URL indicating that the second web page was not accessed.

5. One or more non-transitory computer-readable media comprising instructions that, when executed with one or more processors, cause a system to at least:

receive, from a computing system, a request for information;

provide, to the computing system, a first network-based document of a network-based resource, the first network-based document comprising a portion of the information, code, and an identifier of a second network-based document of the network-based resource, the code comprising statements of a programmatic scripting language in accordance with an ECMAScript standard and configured to, upon execution;

determine a presence or an absence of the identifier in a history stored at the computing system and, if the identifier is present in the history, one or more of;

a state or a style attribute of the identifier from the history, and

determine whether the second network-based document was accessed prior to providing the first network-based document to the computing system;

determine an indication whether the second network-based document was accessed, the indication determined, upon an execution of the code at the computing system, based at least in part on a determination of the presence or the absence of the identifier in the history or on a determination of the state or style attribute of the identifier if the identifier is present in the history; and

determine, based at least in part on the indication, that the request for the information is associated with a network crawler hosted on the computing system.

6. The one or more non-transitory computer-readable media of claim 5, wherein the first network-based document and the second network-based document are associated with an electronic marketplace, wherein the first network-based document comprises a first web page describing an item, wherein the second network-based document comprises a login web page configured to facilitate an authentication of a user account based at least in part on a username and password, wherein the second network-based document comprises a second web page accessed based at least in part on the authentication of the user account, and wherein the indication indicates that the second web page was not accessed based at least in part on the absence of the identifier from the history.

7. The one or more non-transitory computer-readable media of claim 5, wherein the code is configured to, upon execution, generate the history in association with an application of the computing system, wherein the application is configured to request the information.

8. The one or more non-transitory computer-readable media of claim 7, wherein the indication indicates that the second network-based document was not accessed based at least in part on the history lacking the identifier of the second network-based document.

9. The one or more non-transitory computer-readable media of claim 5, wherein the identifier represents a link, and wherein the statements are configured to generate the indication based at least in part on one or more of:

the state or the style attribute of the link from the history.

10. The one or more non-transitory computer-readable media of claim 5, wherein the instructions when executed with the one or more processors further cause the system to at least:

provide the portion of the information from a source selected based at least in part on determining that the request is associated with a network crawler.

11. The one or more non-transitory computer-readable media of claim 5, wherein the instructions when executed with the one or more processors further cause the system to at least:

provide a subset of the information in response to the request for the information, the subset selected based at least in part on determining that the request is associated with a network crawler.

12. The one or more non-transitory computer-readable media of claim 5, wherein the identifier of the second network-based document is inserted in markup language of the first network-based document such that, upon rendering the first network-based document at the computing system, the identifier and the second network-based document are not rendered to a user.

13. A system comprising:

one or more processors;

one or more computer-readable media comprising instructions that, when executed with the one or more processors, cause the system to at least;

receive, from a computing system, a request for information;

add, to a first network-based document comprising the information, code, and an identifier of a second network-based document, the first network-document and the second network-document associated with a same network-based resource of a provider, the code comprising statements of a programmatic scripting language in accordance with an ECMAScript standard and configured to, upon execution;

determine a presence or an absence of the identifier in a history stored at the computing system and, if the identifier is present in the history, one or more of;

a state or a style attribute of the identifier from the history, and

determine whether the second network-based document was accessed prior to providing the first network-based document to the computing system;

provide the first network-based document to the computing system based at least in part on the request;

receive, from the computing system, an indication whether the second network-based document was accessed, the indication received based at least in part on a determination, upon an execution of the code at the computing system, of the presence or the absence of the identifier in the history or on a determination of the state or style attribute of the identifier if the identifier is present in the browser history; and

determine, based at least in part on the indication, that the request for the information is associated with a network crawler hosted on the computing system.

14. The system of claim 13, wherein the second network-based document is unique to the network crawler, and wherein the indication indicates that the second network-based document was accessed.

15. The system of claim 13, wherein the second network-based document is a part of a botnet honeypot, and wherein the indication indicates that the second network-based document was accessed based at least in part on the presence of the identifier in the history.

16. The system of claim 13, wherein the identifier of the second network-based document is added to the first network-based document based at least in part on likelihood that the second network-based document was accessed by the network crawler exceeding a threshold, and wherein the indication indicates that the second network-based document was accessed based at least in part on the presence of the identifier in the history.

17. The system of claim 13, wherein the identifier of the second network-based document is added to the first network-based document based at least in part on likelihood that the second network-based document was accessed by the network crawler falling below a threshold, and wherein the indication indicates that the second network-based document was not accessed based at least in part on the absence of the identifier from the history.

18. The system of claim 13, wherein the instructions when executed with the one or more processors further cause the system to at least:

provide, to the computing system, an additional identifier of a third network-based document based on the request being associated with the network crawler, wherein the additional identifier is configured to be rendered as an image; and

detecting that the third network-based document was not accessed based at least in part on a determination that the image was not rendered.

19. The system of claim 13, wherein the second network-based document is associated with likelihood of being accessed exceeding a threshold, wherein the code is configured to execute after a time delay associated with rendering the first network-based document, and wherein the indication indicates that the second network-based document was accessed after an elapse of the time delay.

20. The system of claim 13, wherein the second network-based document comprises a web page belonging to a portion of a web site, wherein the portion is indicated as inaccessible to the web crawler in a robots exclusion protocol of the web site, and wherein the indication indicates that the web page was not accessed.