Real-time identification of URLs accessed by automated processes

• US 8,903,986 B1
• Filed: 04/05/2010
• Issued: 12/02/2014
• Est. Priority Date: 04/05/2010
• Status: Active Grant

• Current Assignee: NortonLifeLock Inc.
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Newstadt, Keith, Schepis, Adam
• Primary Examiner(s): Kim, Edward
• Application Number:
  US12/754,378
• Time in Patent Office: 1,702 Days
• Field of Search: 709/217, 709/219, 709223-225
• US Class Current: 709/224
• CPC Class Codes: H04L 63/0236 Filtering by address, protocol, port number or service, e.g. IP-address or URL
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

Outbound network traffic originating from a client computer is monitored, and requests to access URLs in the outbound network traffic are detected. Detected requests to access URLs are compared to each of a plurality of rules, in order to determine whether specific ones of the detected requests were initiated automatically by a process or manually by a human user. Where a rule of the plurality applies to a detected request, a rating concerning the request is adjusted. The rating indicates a likelihood as to whether the request was initiated automatically or manually. Depending upon whether the rating concerning the request reaches a threshold, the request is categorized as being either automatically initiated by a process, or manually initiated by a human user. Only those requests categorized as being manually initiated by a human user are logged.

Claims

1. A computer implemented method for identifying requests to access URLs that are made automatically by processes, the method comprising the steps of:

monitoring, by a computer, outbound network traffic originating from a client computer;

detecting requests to access URLs in the outbound network traffic, by a computer;

comparing a plurality of rules to detected requests to access URLs, by a computer, in order to determine whether specific ones of the detected requests were initiated automatically by at least one computerized process as opposed to manually by a human user;

responsive to a rule of the plurality applying to a specific detected request, adjusting a rating concerning the specific detected request, by a computer, the rating indicating a likelihood as to whether the specific detected request was initiated automatically;

categorizing, by a computer, some detected requests to access URLs as being initiated automatically, responsive to an associated rating indicating a likelihood as to whether the request was initiated automatically;

categorizing, by a computer, some detected requests to access URLs as being initiated manually by a human user, responsive to an associated rating indicating a likelihood as to whether the request was initiated automatically; and

at client side, logging, by a computer, only human user activity of interest comprising requests categorized as being manually initiated by the human user, while avoiding logging of information that is superfluous to monitoring of the human user.

2. The method of claim 1 further comprising:

comparing each rule of the plurality to a specific detected request to access a URL, by a computer, in order to determine whether the specific detected request was initiated automatically;

wherein adjusting a rating concerning the specific detected request comprises;

for each rule of the plurality that applies to the specific detected request, adjusting the rating concerning the specific detected request, by a computer, the rating indicating a likelihood that the specific detected request was initiated automatically.

3. The method of claim 2 further comprising:

categorizing, by a computer, the specific detected request as being initiated automatically by a process, responsive to the associated rating reaching a threshold.

4. The method of claim 2 further comprising:

categorizing, by a computer, the specific detected request as being initiated manually by a human user, responsive to a relationship between the associated rating and a threshold.

5. The method of claim 1 wherein adjusting the rating concerning the specific detected request, by a computer, the rating indicating a likelihood as to whether the request was initiated automatically further comprises:

adjusting, by a computer, the rating concerning the specific detected request responsive to a rate at which the URL of the specific detected request is being accessed over time.

6. The method of claim 1 wherein adjusting the rating concerning the specific detected request, by a computer, the rating indicating a likelihood as to whether the request was initiated automatically further comprises:

adjusting, by a computer, the rating concerning the specific detected request responsive to a number of URLs that have been accessed within a given time period by the client computer.

7. The method of claim 1 wherein adjusting the rating concerning the specific detected request, by a computer, the rating indicating a likelihood as to whether the request was initiated automatically further comprises:

adjusting, by a computer, the rating concerning the specific detected request responsive to a human user of the client computer being idle at the time of the specific detected request.

8. The method of claim 1 wherein adjusting the rating concerning the specific detected request, by a computer, the rating indicating a likelihood as to whether the request was initiated automatically further comprises:

adjusting, by a computer, the rating concerning the specific detected request based on a source of the specific detected request.

9. The method of claim 1 wherein adjusting the rating concerning the specific detected request, by a computer, the rating indicating a likelihood as to whether the request was initiated automatically further comprises:

adjusting, by a computer, the rating concerning the specific detected request responsive to a response to the specific detected request.

10. The method of claim 1 further comprising:

responsive to categorizing a detected request as being initiated automatically by a process, transmitting, by a computer, the detected request to a central server for analysis.

11. The method of claim 1 further comprising:

modifying at least one rule of the plurality, by a computer, in response to at least one pattern in requests to access URLs detected over a period of time.

12. The method of claim 1 further comprising:

automatically generating at least one rule of the plurality of rules, by a computer, responsive to data concerning categorized requests amalgamated from a plurality of client computers.

13. The method of claim 1 further comprising:

automatically generating a list of categorized URLs, by a computer, responsive to data concerning categorized requests amalgamated from a plurality of client computers.

14. At least one computer readable storage medium storing a computer program product for identifying requests to access URLs that are made automatically by processes, the computer program product comprising:

program code for monitoring outbound network traffic originating from a client computer;

program code for detecting requests to access URLs in the outbound network traffic;

program code for comparing a plurality of rules to detected requests to access URLs, in order to determine whether specific ones of the detected requests were initiated automatically by at least one computerized process as opposed to manually by a human user;

program code for, responsive to a rule of the plurality applying to a specific detected request, adjusting a rating concerning the specific detected request the rating indicating a likelihood as to whether the specific detected request was initiated automatically;

program code for categorizing some detected requests to access URLs as being initiated automatically, responsive to an associated rating indicating a likelihood as to whether the request was initiated automatically;

program code for categorizing some detected requests to access URLs as being initiated manually by a human user, responsive to an associated rating indicating a likelihood as to whether the request was initiated automatically; and

program code for, at client side, logging only human user activity of interest comprising requests categorized as being manually initiated by the human user, while avoiding logging of information that is superfluous to monitoring of the human user.

15. The computer program product of claim 14 further comprising:

program code for comparing each rule of the plurality to a specific detected request to access a URL, in order to determine whether the specific detected request was initiated automatically;

program code for adjusting, for each rule of the plurality that applies to the specific detected request, the rating concerning the specific detected request the rating indicating a likelihood that the specific detected request was initiated automatically;

program code for categorizing the specific detected request as being initiated automatically by a process, responsive to the associated rating reaching a threshold; and

program code for categorizing the specific detected request as being initiated manually by a human user, responsive to a relationship between the associated rating and a threshold.

16. The computer program product of claim 14 wherein the program code for adjusting the rating concerning the specific detected request further comprises program code for adjusting the rating concerning the specific detected request responsive to at least one factor from a group of factors consisting of:

a rate at which the URL of the specific detected request is being accessed over time;

a number of URLs that have been accessed within a given time period by the client computer;

a human user of the client computer being idle at the time of the specific detected request;

a source of the specific detected request; and

a response to the specific detected request.

17. The computer program product of claim 14 further comprising:

program code for, responsive to categorizing a detected request as being initiated automatically by a process, transmitting the detected request to a central server for analysis.

18. The computer program product of claim 14 further comprising:

program code for modifying at least one rule of the plurality, in response to at least one pattern in requests to access URLs detected over a period of time.