Malware data item analysis

• US 9,021,260 B1
• Filed: 08/29/2014
• Issued: 04/28/2015
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Lim, Robin, Glazer, Michael, Falk, Matthew, Yousaf, Timothy, Staehle, Joseph, Lemanowicz, Lucas, Noury, Sebastien
• Primary Examiner(s): Armouche, Hadi
• Assistant Examiner(s): Zarrineh, Shahriar
• Application Number:
  US14/473,860
• Time in Patent Office: 242 Days
• Field of Search: 713/165
• US Class Current: 713/165
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06F 2221/2107 File encryption

Abstract

Embodiments of the present disclosure relate to a data analysis system that may automatically analyze a suspected malware file, or group of files. Automatic analysis of the suspected malware file(s) may include one or more automatic analysis techniques. Automatic analysis of may include production and gathering of various items of information related to the suspected malware file(s) including, for example, calculated hashes, file properties, academic analysis information, file execution information, third-party analysis information, and/or the like. The analysis information may be automatically associated with the suspected malware file(s), and a user interface may be generated in which the various analysis information items are presented to a human analyst such that the analyst may quickly and efficiently evaluate the suspected malware file(s). For example, the analyst may quickly determine one or more characteristics of the suspected malware file(s), whether or not the file(s) is malware, and/or a threat level of the file(s).

Claims

1. A computer system comprising:

one or more computer readable storage devices configured to store;

a plurality of computer executable instructions;

a plurality of file data items and submission data items, each submission data item associated with at least one file data item, each file data item comprising a suspected malware data item, each submission data item further including indications of at least;

a filename of an associated file data item that was submitted,

a date the associated file data item was submitted, and

an identifier of the person who submitted the associated file data item; and

a graph comprising nodes and edges, each of the nodes representing at least one of a file data item, a submission data item, an analysis data item, or another type of data item, each of the edges indicating an association between two of the nodes; and

one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the computer system to automatically;

in response to receiving a first file data item;

determine whether the received first file data item was previously received by comparing the received first file data item to the plurality of file data items; and

generate a first submission data item;

in response to determining that the first file data item was not previously received;

initiate an analysis of the first file data item, wherein the analysis of the first file data item generates analysis information items, wherein initiating the analysis of the first file data item comprises;

initiating an internal analysis of the first file data item including at least calculation of a hash of the file data item; and

initiating an external analysis of the first file data item by one or more third party analysis systems;

associate the analysis information items with the first file data item; and

associate the first submission data item with the first file data item;

in response to receiving a second file data item;

determine whether the received second file data item was previously received by comparing the received second file data item to the plurality of file data items; and

generate a second submission data item;

in response to determining that the second file data item matches the first data item that was previously received, associate the second submission data item with the first file data item that was previously received; and

generate a user interface including one or more user selectable portions presenting various of the analysis information items associated with the first file data item, the user interface useable by an analyst to determine one or more characteristics of the first file data item, the one or more user selectable portions including a first selectable element, the first selectable element configured to cause, in response to an analyst input selecting the first selectable element, a generation of a graphical visualization including at least;

a first graphical representation of a first node representing the first file data item,

a second graphical representation of a second node representing the first submission data item,

a third graphical representation of an edge connecting the first and second graphical representations and representing the association between the first file data item and the first submission data item,

a fourth graphical representation of a third node representing the second submission data item, and

a fifth graphical representation of a second edge connecting the first and fourth graphical representations and representing the association between the first file data item and the second submission data item.

2. The computer system of claim 1, wherein, further in response to determining that the second file data item matches the first data item that was previously received, the analyst is notified via the user interface that the second file data item was previously received.

3. The computer system of claim 1, wherein determining whether the received first file data item was previously received comprises:

calculating a hash of the received first file data item and comparing the calculated hash to previously calculated hashes associated with the plurality of file data items.

4. The computer system of claim 1, wherein the internal analysis includes analysis performed by the one or more hardware computer processors, and wherein the internal analysis further includes at least one of calculation of an MD5 hash of the first file data item, calculation of a SHA-1 hash of the first file data item, calculation of a SHA-256 hash of the first file data item, calculation of an SSDeep hash of the first file data item, or calculation of a size of the first file data item.

5. The computer system of claim 1, wherein the external analysis includes analysis performed by at least a second computer system, and wherein the external analysis includes execution of the first file data item in a sandboxed environment and analysis of the first file data item by a third-party malware analysis service.

6. The computer system of claim 5, wherein any payload provided by the first file data item after execution of the first file data item in the sandboxed environment is associated with the first file data item as an analysis information item.

7. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions in order to cause the computer system to:

in response to an analyst input, sharing the first file data item and associated analysis information items with a second computer system via a third computer system.

15. The computer system of claim 1, wherein the updated graphical visualization further includes at least:

a sixth graphical representation of a fourth node representing one of the analysis information items, and

a seventh graphical representation of a third edge connecting the first and sixth graphical representations and representing the association between the first file data item and the one of the analysis information items.

8. A computer-implemented method comprising:

storing on one or more computer readable storage devices;

a plurality of computer executable instructions;

a plurality of file data items and submission data items, each submission data item associated with at least one file data item, each file data item comprising a suspected malware data item, each submission data item further including indications of at least;

a filename of an associated file data item that was submitted,

a date the associated file data item was submitted, and

an identifier of the person who submitted the associated file data item; and

a graph comprising nodes and edges, each of the nodes representing at least one of a file data item, a submission data item, an analysis data item, or another type of data item, each of the edges indicating an association between two of the nodes;

in response to receiving a first file data item;

determining, by one or more hardware computer devices configured with specific computer executable instructions, whether the received first file data item was previously received by comparing the received first file data item to the plurality of file data items; and

generating, by the one or more hardware computer devices, a first submission data item; and

in response to determining that the first file data item was not previously received;

initiating, by the one or more hardware computer devices, an analysis of the first file data item, wherein the analysis of the first file data item generates analysis information items, wherein the initiating analysis of the first file data item comprises initiating an internal analysis of the first file data item including at least calculation of a hash of the file data item;

associating, by the one or more hardware computer devices, the analysis information items with the first file data item; and

associating, by the one or more hardware computer devices, the first submission data item with the first file data item;

in response to receiving a second file data item;

determining, by the one or more hardware computer devices, whether the received second file data item was previously received by comparing the received second file data item to the plurality of file data items; and

generating, by the one or more hardware computer devices, a second submission data item;

in response to determining that the second file data item matches the first data item that was previously received, associating, by the one or more hardware computer devices, the second submission data item with the first file data item that was previously received; and

generating, by the one or more hardware computer devices, a user interface including one or more user selectable portions presenting various of the analysis information items associated with the first file data item, the user interface usable by an analyst to determine one or more characteristics of the first file data item, the one or more user selectable portions including a first selectable element, the first selectable element configured to cause, in response to an analyst input selecting the first selectable element, a generation of a graphical visualization including at least;

a first graphical representation of a first node representing the first file data item,

a second graphical representation of a second node representing the first submission data item,

a third graphical representation of an edge connecting the first and second graphical representations and representing the association between the first file data item and the first submission data item,

a fourth graphical representation of a third node representing the second submission data item, and

a fifth graphical representation of a second edge connecting the first and fourth graphical representations and representing the association between the first file data item and the second submission data item.

9. The computer-implemented method of claim 8, wherein, further in response to determining that the second file data item matches the first data item that was previously received, the analyst is notified via the user interface that the second file data item was previously received.

10. The computer-implemented method of claim 8, wherein the internal analysis includes analysis performed by the one or more hardware computer processors, and wherein the internal analysis further includes at least one of calculation of an MD5 hash of the first file data item, calculation of a SHA-1 hash of the first file data item, calculation of a SHA-256 hash of the first file data item, calculation of an SSDeep hash of the first file data item, or calculation of a size of the first file data item.

16. The computer-implemented method of claim 8, wherein the updated graphical visualization further includes at least:

a sixth graphical representation of a fourth node representing one of the analysis information items, and

a seventh graphical representation of a third edge connecting the first and sixth graphical representations and representing the association between the first file data item and the one of the analysis information items.

11. A non-transitory computer-readable storage medium storing software instructions that, in response to execution by a computer system having one or more hardware processors, configure the computer system to perform operations comprising:

storing on one or more computer readable storage devices;

a plurality of computer executable instructions;

a plurality of file data items and submission data items, each submission data item associated with at least one file data item, each file data item comprising a suspected malware data item, each submission data item further including indications of at least;

a filename of an associated file data item that was submitted,

a date the associated file data item was submitted, and

an identifier of the person who submitted the associated file data item; and

a graph comprising nodes and edges, each of the nodes representing at least one of a file data item, a submission data item, an analysis data item, or another type of data item, each of the edges indicating an association between two of the nodes;

in response to receiving a first file data item;

determining whether the received first file data item was previously received by comparing the received first file data item to the plurality of file data items; and

generating a first submission data item; and

in response to determining that the first file data item was not previously received;

initiating an analysis of the first file data item, wherein the analysis of the first file data item generates analysis information items, wherein the initiating analysis of the first file data item comprises initiating an internal analysis of the first file data item including at least calculation of a hash of the file data item;

associating the analysis information items with the first file data item; and

associating the first submission data item with the first file data item;

in response to receiving a second file data item;

determining whether the received second file data item was previously received by comparing the received second file data item to the plurality of file data items; and

generating a second submission data item;

in response to determining that the second file data item matches the first data item that was previously received, associating the second submission data item with the first file data item that was previously received; and

generating a user interface including one or more user selectable portions presenting various of the analysis information items associated with the first file data item, the user interface usable by an analyst to determine one or more characteristics of the first file data item, the one or more user selectable portions including a first selectable element, the first selectable element configured to cause, in response to an analyst input selecting the first selectable element, a generation of a graphical visualization including at least;

a first graphical representation of a first node representing the first file data item,

a second graphical representation of a second node representing the first submission data item,

a third graphical representation of an edge connecting the first and second graphical representations and representing the association between the first file data item and the first submission data item,

a fourth graphical representation of a third node representing the second submission data item, and

a fifth graphical representation of a second edge connecting the first and fourth graphical representations and representing the association between the first file data item and the second submission data item.

12. The non-transitory computer-readable storage medium of claim 11, wherein, further in response to determining that the second file data item matches the first data item that was previously received, the analyst is notified via the user interface that the second file data item was previously received.

13. The non-transitory computer-readable storage medium of claim 11, wherein the internal analysis includes analysis performed by the one or more hardware computer processors, and wherein the internal analysis further includes at least one of calculation of an MD5 hash of the first file data item, calculation of a SHA-1 hash of the first file data item, calculation of a SHA-256 hash of the first file data item, calculation of an SSDeep hash of the first file data item, or calculation of a size of the first file data item.

14. The non-transitory computer-readable storage medium of claim 11, wherein the initiating analysis of the first file data item further comprises:

initiating an external analysis of the first file data item,

wherein the external analysis includes execution of the first file data item in a sandboxed environment and analysis of the first file data item by a third-party malware analysis service.

17. The non-transitory computer-readable storage medium of claim 11, wherein the updated graphical visualization further includes at least:

a sixth graphical representation of a fourth node representing one of the analysis information items, and

a seventh graphical representation of a third edge connecting the first and sixth graphical representations and representing the association between the first file data item and the one of the analysis information items.