Malicious software detection in a computing system

• US 9,043,894 B1
• Filed: 02/06/2015
• Issued: 05/26/2015
• Est. Priority Date: 11/06/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Stowe, Geoff, Dennison, Drew, Anderson, Adam
• Primary Examiner(s): SHAIFER HARRIMAN, DANT B
• Application Number:
  US14/616,080
• Time in Patent Office: 109 Days
• Field of Search: 726/11
• US Class Current: 726/11
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A computer system identifies malicious Uniform Resource Locator (URL) data items from a plurality of unscreened data items that have not been previously identified as associated with malicious URLs. The system can execute a number of pre-filters to identify a subset of URLs in the plurality of data items that are likely to be malicious. A scoring processor can score the subset of URLs based on a plurality of input vectors using a suitable machine learning model. Optionally, the system can execute one or more post-filters on the score data to identify data items of interest. Such data items can be fed back into the system to improve machine learning or can be used to provide a notification that a particular resource within a local network is infected with malicious software.

Claims

1. A computer system to identify malicious Uniform Resource Locator (URL) data items from a plurality of unscreened data items that have not been previously identified as associated with malicious URLs, the system comprising:

one or more computer readable storage devices configured to store

one or more software modules including computer executable instructions; and

the plurality of unscreened data items associated with communications between computerized devices within a local network and external resources, the unscreened data items comprising a plurality of device identifiers for the computerized devices and a plurality of URLs referencing the external resources;

a network connection configured to access, from a remote network not within the local network, a list of domain names satisfying a ranking condition based on Internet traffic data; and

one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the one or more software modules in order to cause the computer system to;

access, from the one or more computer readable storage devices, the plurality of unscreened data items;

identify, from the plurality of unscreened data items, a plurality of connection records, each of the connection records indicating a communication from a computerized device to an external resource at a specific time, such that each of the connection records is associated with a device identifier and a URL,

identify, from the plurality of connection records, one or more connection records having a common device identifier, the identified one or more connection records associated with one or more URLs;

parse the one or more URLs for one or more domain names, each of the one or more URLs associated with a domain name;

based on a determination that none of the one or more domain names satisfies a threshold position in the list of domain names, designate the one or more URLs as possible malicious URL data items;

assign a score based on a plurality of factors relating to the possible malicious URL data items, the factors comprising the determination that none of the one or more domain names satisfies the threshold position in the list of domain names.

2. The system of claim 1, the plurality of unscreened data items comprising a plurality of beaconing malware-related data items and the one or more hardware computer processors further configured to execute the one or more software modules in order to cause the computer system to

access, from the one or more computer readable storage devices, the plurality of beaconing malware-related data items;

generate, based on the accessed beaconing malware-related data items, a plurality of connection pairs, each of the connection pairs indicating communications between an internal source within the local network and an external destination that is not within the local network;

identify a plurality of connection pairs having a common internal source and a common external destination;

generate a time series of connection pairs based on the identified plurality of connection pairs;

filter out noise from the at least one time series to generate a filtered at least one time series;

compute a variance in the filtered at least one time series; and

based on a determination that the variance satisfies a threshold;

designate a connection pair associated with the filtered at least one time series as a seed, the designated connection pair including the common internal source and the common external source;

generate a data item cluster based on the designated seed, wherein generating the data item cluster comprises;

adding the designated seed to the data item cluster;

accessing, from the one or more computer readable storage devices, the clustering strategy; and

adding to the data item cluster, based on the clustering strategy, one or more beaconing malware-related data items determined to be associated with the designated seed; and

score the generated data item cluster, the factors comprising the data item cluster score.

3. The system of claim 1, the one or more computer readable storage devices configured to store a plurality of domain names associated with URLs in communications from computerized devices within a local network from a period of time, and the one or more hardware computer processors further configured to execute the one or more software modules in order to cause the computer system to

access, from the one or more computer readable storage devices, the plurality of domain names;

based on a determination that none of the one or more domain names is included in the plurality of domain names, designate the one or more URLs as possible malicious URL data items,

the factors comprising the determination that none of the one or more domain names is included in the plurality of domain names.

4. The system of claim 1, the one or more computer readable storage devices configured to store a plurality of dictionary words, and the one or more hardware computer processors further configured to execute the one or more software modules in order to cause the computer system to

access, from the one or more computer readable storage devices, the plurality of dictionary words;

based on a determination that none of the one or more domain names is included in the plurality of dictionary words, designate the one or more URLs as possible malicious URL data items,

the factors comprising the determination that none of the one or more domain names is included in the plurality of dictionary words.

5. The system of claim 1, the one or more computer readable storage devices configured to store a plurality of filepaths associated with URLs in communications from computerized devices within a local network from a period of time, and the one or more hardware computer processors further configured to execute the one or more software modules in order to cause the computer system to

access, from the one or more computer readable storage devices, the plurality of filepaths;

parse a URL for an associated filepath;

based on a determination that the filepath is included in the plurality of filepaths, designate the URL as a possible malicious URL data item,

the factors comprising the determination that the filepath is included in the plurality of filepaths.

6. The system of claim 1,

the one or more computer readable storage devices configured to

store a distribution of n-grams for filepaths associated with a domain name having a rank indicating that the domain name is associated with a amount of Internet traffic, and

store a second distribution of n-grams for filepaths associated with the domain name;

and

the one or more hardware computer processors further configured to execute the one or more software modules in order to cause the computer system to compare the expected distribution of n-grams to the actual distribution of n-grams, the factors comprising a variance between the distributions.

7. The system of claim 1,

the network connection configured to access, from a remote network not within the local network, an Internet search engine providing an autocomplete function that automatically displays words to complete a query entered into the search engine and to receive from the remote network the words suggested by the autocomplete function,

the one or more computer readable storage devices configured to store a list of words associated with malicious software;

and

the one or more hardware computer processors further configured to execute the one or more software modules in order to cause the computer system to

transmit to the Internet search engine a query comprising a domain name associated with a URL, and

receive words displayed by the search engine in response to the query,

the factors comprising the received words that are also included in the list of words.

8. The system of claim 1,

the network connection configured to access, from a remote network not within the local network, an Internet service providing WHOIS and/or DNS registration data to receive from the remote network domain registration data,

the one or more hardware computer processors further configured to execute the one or more software modules in order to cause the computer system to

transmit to the Internet search engine a query comprising a domain name associated with a URL, and

receive a domain registration date in response to the query;

the factors comprising the received domain registration date.

9. The system of claim 1, the score based on a Support Vector Machine model, a Neural Network model, a Decision Tree model, a Naï

ve Bayes model, or a Logistic Regression model.