Malicious software detection in a computing system

• US 10,728,277 B2
• Filed: 10/01/2018
• Issued: 07/28/2020
• Est. Priority Date: 11/06/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Dennison, Drew, Stowe, Geoff, Anderson, Adam
• Primary Examiner(s): Shaifer Harriman, Dant B
• Application Number:
  US16/148,241
• Publication Number:
  US 20190036945A1
• Time in Patent Office: 666 Days
• Field of Search: 726 23
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A computer system identifies malicious Uniform Resource Locator (URL) data items from a plurality of unscreened data items that have not been previously identified as associated with malicious URLs. The system can execute a number of pre-filters to identify a subset of URLs in the plurality of data items that are likely to be malicious. A scoring processor can score the subset of URLs based on a plurality of input vectors using a suitable machine learning model. Optionally, the system can execute one or more post-filters on the score data to identify data items of interest. Such data items can be fed back into the system to improve machine learning or can be used to provide a notification that a particular resource within a local network is infected with malicious software.

Claims

1. A computer system for detecting malicious software, the computer system comprising:

one or more computer-readable storage devices including computer executable instructions; and

one or more processors configured to execute the computer executable instructions in order to;

access connection records that include respective locational references to computerized resources external to a local network which computerized devices within the local network have accessed or attempted to access;

perform one or more filtering operations on the connection records, wherein the one or more filtering operations include;

identifying, within the connection records, a first subset of the connection records associated with a first locational reference;

determining first n-gram distribution data indicating a frequency of each n-length character combination that occurs as an n-length window moves across at least parts of one or more filepaths associated with the first subset of the connection records, the n-lengths being two or more characters;

accessing second n-gram distribution data representing an expected n-gram distribution associated with likely non-malicious locational references;

comparing the first n-gram distribution data with the second n-gram distribution data to determine a variance; and

at least in part in response to the variance exceeding a threshold, transmitting an indicator for display, the indicator indicating the first locational reference is likely to compromise security;

score at least some of the first subset of the connection records using a machine learning model incorporating a factor relating to the first locational reference; and

perform one or more additional filtering operations on the scored first subset of the connection records to identify a second subset of the scored first subset of the connection records,

wherein the one or more additional filtering operations identify, within the scored first subset of the connection records, the second subset of the scored first subset of the connection records, and the second subset of the scored first subset of the connection records are more likely to be malicious than locational references associated with connection records in the scored first subset of the connection records that are not included in the second subset of the scored first subset of the connection records.

2. The computer system of claim 1, wherein the machine learning model incorporates a plurality of factors based on at least one of the one or more filtering operations.

3. The computer system of claim 1, wherein the machine learning model comprises at least one of:

a Support Vector Machine model, a Neural Network model, a Decision Tree model, a Naï

ve Bayes model, or a Logistic Regression model.

4. The computer system of claim 1, wherein the one or more filtering operations includes a filtering operation comprising:

parsing the respective locational references associated with a certain connection record in the connection records for a domain name; and

based on a determination that the domain name does not satisfy a threshold position in a list of domain names satisfying a ranking condition based on Internet traffic data, identifying the certain connection record to be in the first subset of the connection records.

5. The computer system of claim 1, wherein the one or more filtering operations includes a filtering operation comprising:

parsing the respective locational references associated with a certain connection record in the connection records for a domain name; and

based on a determination that the domain name is not included in a set of domain names associated with a set of locational references in a set of communications involving the local network from a period of time, identifying the certain connection record to be in the first subset of the connection records.

6. The computer system of claim 1, wherein the one or more filtering operations includes a filtering operation comprising:

parsing the respective locational references associated with a certain connection record in the connection records for a domain name; and

based on a determination that the domain name is not included in a plurality of dictionary words, identifying the certain connection record to be in the first subset of the connection records.

7. The computer system of claim 1, wherein the one or more filtering operations includes a filtering operation comprising:

parsing the respective locational references associated with a certain connection record in the connection records for a filepath; and

based on a determination that the filepath is in a plurality of filepaths associated with a set of locational references in a set of communications involving the local network from a period of time, identifying the certain connection record to be in the first subset of the connection records.

8. The computer system of claim 1, wherein determining the first n-gram distribution data includes:

parsing the first subset of the connection records to identify the one or more filepaths;

determining the frequency of each n-length character combination identified in one or more filepaths; and

generating the first n-gram distribution data based on how frequently each n-gram of the plurality of n-grams occurs in the one or more filepaths.

9. The computer system of claim 1, wherein the one or more filtering operations includes a filtering operation comprising:

parsing the respective locational references associated with a certain connection record in the connection records for a domain name;

accessing a list of words associated with malicious locational references;

transmitting, to an Internet search engine providing an autocomplete function that automatically displays words to complete a query entered into the Internet search engine, a first query comprising the domain name;

receiving, from the Internet search engine, the words displayed in response to the first query; and

based on a determination that at least one of the words is in a list of words associated with malicious locational references, identifying the certain connection record to be in the first subset of the connection records.

10. The computer system of claim 1, wherein the one or more filtering operations includes a filtering operation comprising:

parsing the respective locational references associated with a certain connection record in the connection records for a domain name; and

based on a registration date of the domain name, identifying the certain connection record to be in the first subset of the connection records.

18. The computer system of claim 8, wherein the respective locational references comprise domain names, and wherein the one or more filepaths comprises URL information following the domain names.

11. A filtering system for filtering connection records, the filtering system including:

a computer-readable storage device storing computer executable instructions and one or more hardware computer processors configured to execute the computer executable instructions in order to;

access connection records that include respective locational references to computerized resources external to a local network which computer devices within the local network have accessed or attempted to access;

perform one or more filtering operations on the connection records, wherein the one or more filtering operations include;

identifying, within the connection records, a first subset of the connection records associated with a first locational reference;

determining first n-gram distribution data indicating a frequency of each n-length character combination that occurs as an n-length window moves across at least parts of one or more filepaths associated with the first subset of the connection records, the n-length being two or more characters;

accessing second n-gram distribution data indicating frequencies of n-grams in other filepaths associated with non-malicious locational references;

comparing the first n-gram distribution data with the second n-gram distribution data to determine a variance; and

at least in part in response to the variance exceeding a threshold, transmitting an indicator for display, the indicator indicating that the first locational reference is likely to compromise security;

score at least some of the first subset of the connection records using a machine learning model incorporating a factor relating to the first locational reference; and

perform one or more additional filtering operations on the scored first subset of the connection records to identify a second subset of the scored first subset of the connection records,

wherein the one or more additional filtering operations identify, within the scored first subset of the connection records, the second subset of the scored first subset of the connection records, and the second subset of the scored first subset of the connection records are more likely to be malicious than locational references associated with connection records in the scored first subset of the connection records that are not included in the second subset of the scored first subset of the connection records.

12. The filtering system of claim 11, wherein the computer executable instructions further cause the one or more hardware computer processors to:

assign a score to at least some of the first subset of the connection records based on a plurality of factors relating to locational references associated with the first subset of the connection records; and

perform one or more different filtering operations on the scored first subset of the connection records to identify a second subset of the scored first subset of the connection records, wherein the second subset of the scored first subset of the connection records is more likely to be associated with malicious locational references than connection records that are included in the scored first subset of the connection records but are not included in the second subset of the scored first subset of the connection records.

13. The filtering system of claim 11, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational references associated with a certain connection record for a domain name; and

determine whether the domain name satisfies a threshold position in a list of domain names satisfying a ranking condition based on Internet traffic data.

14. The filtering system of claim 11, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational references associated with a certain connection record in the connection records for a domain name; and

determine that the domain name is not included in a set of domain names associated with a set of locational references in a set of communications involving the local network from a period of time.

15. The filtering system of claim 11, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational references associated with a certain connection record in the connection records for a filepath; and

determine that the filepath is in a plurality of filepaths associated with a set of locational references in a set of communications involving the local network from a period of time.

16. The filtering system of claim 11, wherein determining the first n-gram distribution data includes:

parsing the first subset of the connection records to identify the one or more filepaths;

determining the frequency of each n-length character combination identified in one or more filepaths; and

generating the first n-gram distribution data based on how frequently each n-gram of the plurality of n-grams occurs in the one or more filepaths.

17. The filtering system of claim 11, wherein the computer executable instructions further cause the one or more hardware computer processors to:

parse the respective locational references associated with a certain connection record in the connection records for a domain name;

access a list of words associated with malicious locational references;

transmit, to an Internet search engine providing an autocomplete function that automatically displays words to complete a query entered into the Internet search engine, a first query comprising the domain name;

receive, from the Internet search engine, the words displayed in response to the first query; and

determine that at least one of the words is in a list of words associated with malicious locational references.

19. A filtering system for filtering connection records, the filtering system including:

a computer-readable storage device storing computer executable instructions and one or more hardware computer processors configured to execute the computer executable instructions in order to;

access connection records that include respective locational references to computerized resources external to a local network which computer devices within the local network have accessed or attempted to access;

parse the respective locational references associated with a certain connection record in the connection records for a domain name;

determine that the domain name is not included in a plurality of dictionary words; and

perform one or more filtering operations on the connection records, wherein the one or more filtering operations include;

identifying, within the connection records, a first subset of the connection records associated with a first locational reference;

determining first n-gram distribution data indicating a frequency of each n-length character combination that occurs as an n-length window moves across at least parts of one or more filepaths associated with the first subset of the connection records, the n-length being two or more characters;

accessing second n-gram distribution data indicating frequencies of n-grams in other filepaths associated with non-malicious locational references;

comparing the first n-gram distribution data with the second n-gram distribution data to determine a variance; and

at least in part in response to the variance exceeding a threshold, transmitting an indicator for display, the indicator indicating that the first locational reference is likely to compromise security.