Malicious software detection in a computing system

• US 9,558,352 B1
• Filed: 04/28/2015
• Issued: 01/31/2017
• Est. Priority Date: 11/06/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Dennison, Drew, Stowe, Geoff, Anderson, Adam
• Primary Examiner(s): Harriman, Dant Shaifer
• Application Number:
  US14/698,432
• Time in Patent Office: 644 Days
• Field of Search: 726/23
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A computer system identifies malicious Uniform Resource Locator (URL) data items from a plurality of unscreened data items that have not been previously identified as associated with malicious URLs. The system can execute a number of pre-filters to identify a subset of URLs in the plurality of data items that are likely to be malicious. A scoring processor can score the subset of URLs based on a plurality of input vectors using a suitable machine learning model. Optionally, the system can execute one or more post-filters on the score data to identify data items of interest. Such data items can be fed back into the system to improve machine learning or can be used to provide a notification that a particular resource within a local network is infected with malicious software.

Claims

1. A computer-implemented method to identify connection records associated with malicious locational references, the method comprising:

as implemented by one or more computer readable storage devices configured to store one or more software modules including computer executable instructions, and by one or more hardware computer processors in communication with the one or more computer readable storage devices configured to execute the one or more software modules,

identifying connection records, stored in the one or more computer readable storage devices, indicating communications involving a local network, each of the connection records associated with a respective device identifier for a computerized device within the local network, a respective locational reference to a resource external to the local network, and a respective time of communication,

performing one or more filtering operations on the connection records to identify, within the connection records, first connection records more likely to be associated with malicious locational references than connection records not included in the first connection records, such that, once initiated, the one or more filtering operations are performed with the one or more computer readable storage devices and the one or more hardware computer processors and without the need for manually performing the filtering operations;

numerically scoring at least some of the first connection records using a machine learning model incorporating a plurality of factors relating to the locational references associated with the first connection records;

filtering the scored first connection records to identify, within the first connection records, second connection records more likely to be associated with malicious locational references than first connection records not included in the second connection records;

receiving a disposition generated by a user regarding one or more of the second connection records; and

applying the disposition to the machine learning model.

2. The method of claim 1, the plurality of factors based on at least one of the one or more filtering operations.

3. The method of claim 1, the one or more filtering operations comprising a filtering operation including

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name;

and

based on a determination that the domain name does not satisfy a threshold position in a list of domain names satisfying a ranking condition based on Internet traffic data, identifying the certain connection record to be in the first connection records.

4. The method of claim 1, the one or more filtering operations comprising a filtering operation including

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name;

and

based on a determination that the domain name is not included in a set of domain names associated with a set of locational references in a set of communications involving the local network from a period of time, identifying the certain connection record to be in the first connection records.

5. The system of claim 1, the one or more filtering operations comprising a filtering operation including

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name;

and

based on a determination that the domain name is not included in a plurality of dictionary words, identifying the certain connection record to be in the first connection records.

6. The method of claim 1, the one or more filtering operations comprising a filtering operation including

parsing the respective locational reference associated with a certain connection record in the connection records for a filepath;

and

based on a determination that the filepath is in a plurality of filepaths associated with a set of locational references in a set of communications involving the local network from a period of time, identifying the certain connection record to be in the first connection records.

7. The method of claim 1, the one or more filtering operations comprising a filtering operation including

parsing the respective locational reference associated with a certain connection record in the connection records for a parsed domain name;

accessing a first distribution of n-grams for filepaths associated with a predetermined domain name having a rank indicating that the predetermined domain name is associated with an amount of Internet traffic, and

accessing a second distribution of n-grams for filepaths associated with the parsed domain name; and

based on the variance between the first distribution and the second distribution, identifying the certain connection record to be in the first connection records.

8. The method of claim 1, the one or more filtering operations comprising a filtering operation including

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name;

accessing a list of words associated with malicious locational references;

transmitting, to an Internet search engine providing an autocomplete function that automatically displays words to complete a query entered into the Internet search engine, a first query comprising the domain name,

receiving, from the Internet search engine, the words displayed in response to the first query, and

based on a determination that at least one of the words is in a list of words associated with malicious locational references, identifying the certain connection record to be in the first connection records.

9. The method of claim 1, the one or more filtering operations comprising a filtering operation including

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name;

and

based on a registration date of the domain name, identifying the certain connection record to be in the first connection records.

10. The method of claim 1, the machine learning model comprising a Support Vector Machine model, a Neural Network model, a Decision Tree model, a Naï

ve Bayes model, or a Logistic Regression model.

11. A computer-implemented method to identify connection records associated with malicious locational references, the method comprising:

as implemented by one or more computer readable storage devices configured to store one or more software modules including computer executable instructions, and by one or more hardware computer processors in communication with the one or more computer readable storage devices configured to execute the one or more software modules,

identifying connection records, stored in the one or more computer readable storage devices, indicating communications involving a local network, each of the connection records associated with a respective device identifier for a computerized device within the local network, a respective locational reference to a resource external to the local network, and a respective time of communication,

performing one or more filtering operations on the connection records to identify, within the connection records, first connection records more likely to be associated with malicious locational references than connection records not included in the first connection records, such that, once initiated, the one or more filtering operations are performed with the one or more computer readable storage devices and the one or more hardware computer processors and without the need for manually performing the filtering operations;

assigning a score to at least some of the first connection records based on a plurality of factors relating to the locational references associated with the first connection records; and

based on the scoring, generating a notification that a particular connection record of the first connection records is associated with a malicious locational reference.

12. The method of claim 11, the plurality of factors based on at least one of the one or more filtering operations.

13. The method of claim 11, further comprising parsing the respective locational reference associated with a certain connection record for a domain name, the plurality of factors comprising a determination that the domain name does not satisfy a threshold position in a list of domain names satisfying a ranking condition based on Internet traffic data.

14. The method of claim 11, further comprising parsing the respective locational reference associated with a certain connection record in the connection records for a domain name, the plurality of factors comprising a determination that the domain name is not included in a set of domain names associated with a set of locational references in a set of communications involving the local network from a period of time.

15. The method of claim 11, further comprising parsing the respective locational reference associated with a certain connection record in the connection records for a domain name, the plurality of factors comprising a determination that the domain name is not included in a plurality of dictionary words.

16. The method of claim 11, further comprising parsing the respective locational reference associated with a certain connection record in the connection records for a filepath, the plurality of factors comprising a determination that the filepath is in a plurality of filepaths associated with a set of locational references in a set of communications involving the local network from a period of time.

17. The method of claim 11, further comprising

parsing the respective locational reference associated with a certain connection record in the connection records for a parsed domain name;

accessing a first distribution of n-grams for filepaths associated with a predetermined domain name having a rank indicating that the predetermined domain name is associated with an amount of Internet traffic, and

accessing a second distribution of n-grams for filepaths associated with the parsed domain name,

the plurality of factors comprising a determination of the variance between the first distribution and the second distribution.

18. The method of claim 11, further comprising

parsing the respective locational reference associated with a certain connection record in the connection records for a domain name;

accessing a list of words associated with malicious locational references;

transmitting, to an Internet search engine providing an autocomplete function that automatically displays words to complete a query entered into the Internet search engine, a first query comprising the domain name; and

receiving, from the Internet search engine, the words displayed in response to the first query,

the plurality of factors comprising a determination that at least one of the words is in a list of words associated with malicious locational references.

19. The method of claim 11, further comprising parsing the respective locational reference associated with a certain connection record in the connection records for a domain name, the plurality of factors based on a registration date of the domain name.

20. The method of claim 11, wherein assigning the score is based on a Support Vector Machine model, a Neural Network model, a Decision Tree model, a Naï

ve Bayes model, or a Logistic Regression model.