Web Bot detection and human differentiation

• US 10,326,789 B1
• Filed: 09/25/2015
• Issued: 06/18/2019
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Vines, Forrest MacKenzie, Demirjian, Sevag, Scott, Nathan David, Tseng, Jui Te
• Primary Examiner(s): Pham, Luu T
• Assistant Examiner(s): Wilcox, James J
• Application Number:
  US14/865,515
• Time in Patent Office: 1,362 Days
• CPC Class Codes: H04L 2463/144 Detection or countermeasures against botnets
• CPC Class Codes: H04L 63/0281 Proxies
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/1466 Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/563 Data redirection of data network streams
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

Web Bot detection methods and systems are provided that receive a request, in connection with a network session. The methods and systems determine whether the request is associated with potential Bot activity, and based thereon assign a Bot confidence designation. The Bot confidence designation indicates a likelihood that the request represents an agent-based request. The methods and systems analyze a session trait of the network session relative to predetermined session traits indicative of human-based requests, and assign a human confidence designation based on the analysis. The human confidence designation indicates a likelihood that the request represents a human-based request. The request is then classified to represent an agent-based request or human-based request based on the Bot and human confidence designations.

Claims

1. A computer program product comprising a non signal computer readable storage medium comprising computer executable code to perform at least:

obtain, from a computing device and in connection with a network session, a request directed to a network service to be fulfilled by the network service;

compare a request property of interest (POI) representing at least one of a user agent or an Internet Protocol (IP) address from the request to a watch list designating sources of requests associated with computer-implemented automated agents by comparing the at least one of the user agent or the IP address to the watch list;

assign an automated agent confidence designation based on the compare operation, the automated agent confidence designation indicating a likelihood that the request was obtained from a computer-implemented automated agent;

as a result of the automated agent confidence designation being below a predetermined threshold;

analyze a session trait of the network session utilizing a computer model that uses one or more predetermined session traits relative to human confidence designations; and

assign a human confidence designation from the computer model based on the analyze operation, the human confidence designation providing an indication whether the request represents a human-based request, wherein the human-based request classifies the request as submitted via human interaction with a client device;

determine that, based on the automated agent confidence designation and the human confidence designation, the request is an agent-based request, the agent-based request classifies the request as submitted from a computer-implemented automated agent;

determine, as a result of determining that the request is an agent-based request, an action to restrict fulfillment of the request by the network service; and

restrict, by performing the action, fulfillment of the request by the network service limiting the agent-based request from being directed to the network service.

2. The computer program product of claim 1, wherein the analyze operation includes analyzing the session trait from a session history recorded in connection with the network session in search of one or more predetermined session traits associated with human based requests.

3. The computer program product of claim 1, wherein the human confidence designation is based at least in part on one or more of:

a number of webpages visited,

an amount of time spent at a webpage, or

an overall session duration.

4. A computer-implemented method, comprising:

obtaining a request, in connection with a network session, from a computing device to be fulfilled by a network service;

comparing a request property of interest (POI) representing at least one of a user agent or an Internet Protocol (IP) address from the request to a watch list designating sources of requests associated with computer-implemented automated agents, by comparing the at least one of the user agent or the IP address to the watch list;

assigning an automated agent confidence designation based on the comparing operation, the automated agent confidence designation indicating a likelihood that the request was obtained from a computer-implemented automated agent;

in response to the automated agent confidence designation being below a predetermined threshold, analyzing a session trait of the network session utilizing a computer model that uses one or more predetermined session traits relative to human confidence designations to determine an indication whether the request represents an agent-based request;

determining, based at least in part on the automated agent confidence designation and the session trait, that the request is an agent-based request;

determining, as a result of determining that the request is an agent-based request, an action to restrict fulfillment of the request by the network service; and

restricting, by causing performance of the action, fulfillment of the request by the network service by at least limiting the agent-based request from being directed to the network service.

5. The method of claim 4, wherein the action includes declining the request when the request is determined as the agent-based request, and responding to the request when the request is determined as a human-based request.

6. The method of claim 4, wherein determining the request is an agent-based request includes processing first and second requests associated with a common IP address, classifying the first request as the agent-based request and classifying the second request as a human-based request.

7. The method of claim 4, further comprising:

receiving first and second requests in connection with first and second network sessions;

determining that the first and second requests originate from a common IP address associated with potential agent-based activity; and

wherein the analyzing operation identifies the session trait of the first network session to correspond to human-based session behavior, and identifies the session trait of the second network session to not correspond to human-based session behavior.

8. The method of claim 4, further comprising assigning a human confidence designation to the request based on the analyzing operation, the human confidence designation indicating a likelihood that the request represents a human-based request.

9. The method of claim 8, further comprising combining automated agent and human confidence designations resulting in a weighted sum to form traffic classification feedback to be delivered to a proxy service.

10. The method of claim 4, wherein the session trait includes at least one of a number of webpages visited during one or more network sessions, a time spent at a webpage in connection with one or more requests or sessions, an overall duration of a network session, browser patterns followed when navigating through links on a network resource, or transitions between network resources.

11. The method of claim 4, wherein the watch list includes one or more of:

a signature based at least in part on the IP address,

a session associated with the request, or

purchase history associated with the request.

12. A system, comprising:

at least one processor; and

memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to;

obtain, from a computing device directed to a network service and in connection with a network session, a request to be fulfilled by the network service;

compare a request property of interest (POI) representing at least one of a user agent or an Internet Protocol (IP) address from the request to a watch list designating sources of requests associated with computer-implemented automated agents by comparing the at least one of the user agent or the IP address to the watch list;

assign an automated robot confidence designation based on the compare operation, the automated robot confidence designation indicating a likelihood that the request was obtained from a computer implemented agent;

in response to the automated robot confidence designation being below a predetermined threshold, analyze a session trait of the network session utilizing a computer model that uses one or more predetermined session traits relative to human confidence designations to determine an indication whether the request represents an agent-based request;

determine, based at least in part on the automated robot confidence designation and the session trait, that the request is an agent-based request;

determine, as a result of determining that the request is an agent-based request, an action to restrict fulfillment of the request; and

restrict, by causing performance of the action, fulfillment of the request by the network service by at least limiting the agent-based request from being directed to the network service.

13. The system of claim 12, wherein the at least one processor analyzes activity data from a session record in connection with the session trait, the session record stored in the memory and corresponding to the network session, the activity data including at least one of a number of webpages visited, a time spent at each webpage, or an overall session duration.

14. The system of claim 12, wherein the at least one processor processes first and second requests associated with a common IP address, classifies the first request as the agent-based request and classifies the second request as a human-based request.

15. The system of claim 12, wherein the at least one processor:

receives first and second requests in connection with first and second network sessions;

determines that first and second requests originate from a common IP address associated with potential agent-based activity;

identifies the session trait of the first network session to correspond to human based session behavior; and

identifies the session trait of the second network session to not correspond to human-based session behavior.

16. The system of claim 12, wherein the analyze operation includes assigning a human confidence designation to the request, the human confidence designation indicating a likelihood that the request represents a user-based request.

17. The system of claim 12, further comprising at least a first data store storing an activity log of multiple session histories, the analyze operation analyzing one of the session histories associated with the network session to determine whether the session trait is indicative of human-based session behavior.

18. The system of claim 12, further comprising a second data store storing a watch list of IP addresses known to be associated with agent activity, the compare operation comparing an IP address of the request with the watch list to determine whether the request is associated with the computer-implemented automated agents.

19. The system of claim 12, wherein the program instructions are further executable by the at least one processor to decline a second request when the second request is classified as a computer implemented automated agent based request.

20. The system of claim 12, wherein the program instructions further include instructions that are further executable by the at least one processor to:

obtain, from another computing device, another request to be fulfilled by the network service; and

as a result of determining that the other request was submitted by an automated software agent, deny, quarantine, or temporarily suspend the other request.