Malicious software detection in a computing system

• US 10,135,863 B2
• Filed: 12/14/2016
• Issued: 11/20/2018
• Est. Priority Date: 11/06/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Dennison, Drew, Stowe, Geoff, Anderson, Adam
• Primary Examiner(s): Shaifer Harriman, Dant B
• Application Number:
  US15/378,567
• Publication Number:
  US 20170134397A1
• Time in Patent Office: 706 Days
• Field of Search: 726 23
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A computer system identifies malicious Uniform Resource Locator (URL) data items from a plurality of unscreened data items that have not been previously identified as associated with malicious URLs. The system can execute a number of pre-filters to identify a subset of URLs in the plurality of data items that are likely to be malicious. A scoring processor can score the subset of URLs based on a plurality of input vectors using a suitable machine learning model. Optionally, the system can execute one or more post-filters on the score data to identify data items of interest. Such data items can be fed back into the system to improve machine learning or can be used to provide a notification that a particular resource within a local network is infected with malicious software.

Claims

1. A computer system comprising:

one or more computer-readable storage devices including computer executable instructions; and

one or more hardware computer processors configured to execute the computer executable instructions in order to;

identify connection records each associated with a respective device identifier for a computerized device within a local network with an outbound connection to a respective locational reference to a resource external to the local network;

perform one or more filtering operations on the connection records to identify, within the connection records, a first subset of the connection records associated with locational references more likely to be malicious than locational references associated with connection records not included in the first subset of connection records;

score at least some of the first subset of connection records using a machine learning model incorporating a factor relating to the locational references associated with the first subset of connection records; and

perform one or more additional filtering operations on the scored first subset of connection records to identify a second subset of connection records that is fewer in number than the scored first subset of connection records.

2. The computer system of claim 1, wherein the computer executable instructions further cause the one or more processors to:

filter the scored first subset of connection records to identify, within the first subset of connection records, the second subset of connection records associated with locational references that are more likely to be malicious than locational references associated with records in first subset of connection records that are not included in the second subset of connection records.

3. The computer system of claim 1, wherein the machine learning model incorporates a plurality of factors based on at least one of the one or more filtering operations.

4. The computer system of claim 1, wherein the one or more filtering operations comprise a filtering operation including:

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name; and

based on a determination that the domain name does not satisfy a threshold position in a list of domain names satisfying a ranking condition based on Internet traffic data, identifying the certain connection record to be in the first subset of connection records.

5. The computer system of claim 1, wherein the one or more filtering operations comprise a filtering operation including:

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name; and

based on a determination that the domain name is not included in a set of domain names associated with a set of locational references in a set of communications involving the local network from a period of time, identifying the certain connection record to be in the first subset of connection records.

6. The computer system of claim 1, wherein the one or more filtering operations comprise a filtering operation including:

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name; and

based on a determination that the domain name is not included in a plurality of dictionary words, identifying the certain connection record to be in the first subset of connection records.

7. The computer system of claim 1, wherein the one or more filtering operations comprise a filtering operation including:

parsing the respective locational reference associated with a certain connection record in the connection records for a filepath; and

based on a determination that the filepath is in a plurality of filepaths associated with a set of locational references in a set of communications involving the local network from a period of time, identifying the certain connection record to be in the first subset of connection records.

8. The computer system of claim 1, wherein the one or more filtering operations comprise a filtering operation including:

parsing the respective locational reference associated with a certain connection record in the connection records for a parsed domain name;

accessing a first distribution of n-grams for filepaths associated with a predetermined domain name having a rank indicating that the predetermined domain name is associated with an amount of Internet traffic;

accessing a second distribution of n-grams for filepaths associated with the parsed domain name; and

based on a variance between the first distribution and the second distribution, identifying the certain connection record to be in the first subset of connection records.

9. The computer system of claim 1, wherein the one or more filtering operations comprise a filtering operation including:

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name;

accessing a list of words associated with malicious locational references;

transmitting, to an Internet search engine providing an autocomplete function that automatically displays words to complete a query entered into the Internet search engine, a first query comprising the domain name;

receiving, from the Internet search engine, the words displayed in response to the first query; and

based on a determination that at least one of the words is in a list of words associated with malicious locational references, identifying the certain connection record to be in the first subset of connection records.

10. The computer system of claim 1, wherein the one or more filtering operations comprise a filtering operation including:

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name; and

based on a registration date of the domain name, identifying the certain connection record to be in the first subset of connection records.

11. The computer system of claim 1, wherein the machine learning model comprises at least one of:

a Support Vector Machine model, a Neural Network model, a Decision Tree model, a Naï

ve Bayes model, or a Logistic Regression model.

12. A filtering system for filtering connection records, the filtering system including:

a computer-readable storage device storing computer executable instructions and one or more hardware computer processors configured to execute the computer executable instructions in order to;

identify connection records indicating outbound communications each associated with a respective device identifier for a computerized device within the local network outbound to a respective locational reference to a resource external to the local network;

perform one or more filtering operations on the connection records to identify, within the connection records, a first subset of connection records more likely to be associated with malicious locational references than connection records not included in the first subset of connection records;

assign a score to at least some of the first subset of connection records based on a plurality of factors relating to the locational references associated with the first subset of connection records; and

perform one or more different filtering operations on the first scored subset of connection records to identify a second subset of connection records that is fewer in number than the scored first subset of connection records, wherein the second subset of connection records is more likely to be associated with malicious locational references than connection records that are included in the first scored subset of connection records but are not included in the second subset of connection records.

13. The filtering system of claim 12, wherein the plurality of factors are based on at least one of the one or more filtering operations.

14. The filtering system of claim 12, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational reference associated with a certain connection record for a domain name, the plurality of factors comprising a determination that the domain name does not satisfy a threshold position in a list of domain names satisfying a ranking condition based on Internet traffic data.

15. The filtering system of claim 12, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational reference associated with a certain connection record in the connection records for a domain name, the plurality of factors comprising a determination that the domain name is not included in a set of domain names associated with a set of locational references in a set of communications involving the local network from a period of time.

16. The filtering system of claim 12, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational reference associated with a certain connection record in the connection records for a domain name, the plurality of factors comprising a determination that the domain name is not included in a plurality of dictionary words.

17. The filtering system of claim 12, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational reference associated with a certain connection record in the connection records for a filepath, the plurality of factors comprising a determination that the filepath is in a plurality of filepaths associated with a set of locational references in a set of communications involving the local network from a period of time.

18. The filtering system of claim 12, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational reference associated with a certain connection record in the connection records for a parsed domain name;

access a first distribution of n-grams for filepaths associated with a predetermined domain name having a rank indicating that the predetermined domain name is associated with an amount of Internet traffic; and

access a second distribution of n-grams for filepaths associated with the parsed domain name;

wherein the plurality of factors comprise a determination of a variance between the first distribution and the second distribution.

19. The filtering system of claim 12, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational reference associated with a certain connection record in the connection records for a domain name;

access a list of words associated with malicious locational references;

transmit, to an Internet search engine providing an autocomplete function that automatically displays words to complete a query entered into the Internet search engine, a first query comprising the domain name; and

receive, from the Internet search engine, the words displayed in response to the first query, wherein the plurality of factors comprise a determination that at least one of the words is in a list of words associated with malicious locational references.

20. The filtering system of claim 12, wherein assigning the score is based on at least one of:

a Support Vector Machine model, a Neural Network model, a Decision Tree model, a Naï

ve Bayes model, or a Logistic Regression model.