Automatic extraction of indicators of compromise from multiple data sources accessible over a network

• US 10,250,621 B1
• Filed: 11/17/2016
• Issued: 04/02/2019
• Est. Priority Date: 11/17/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Li, Zhou
• Primary Examiner(s): Dinh, Minh
• Application Number:
  US15/354,680
• Time in Patent Office: 866 Days
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9558 Details of hyperlinks; Management of linked annotations
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

A processing device in one embodiment comprises a processor coupled to a memory and is configured to direct one or more web crawlers to obtain textual information from a plurality of data sources accessible over at least one network, to extract terms likely to be associated with indicators of compromise from the obtained textual information, to filter the extracted terms to identify terms corresponding to respective valid indicators of compromise, to generate links between the terms corresponding to the respective valid indicators of compromise, and to convert the links and the corresponding terms into an output document in a specified indicator of compromise format. Feedback from an analyst device receiving the output document may be used to adjust a filter parameter of the extracted term filtering. Additionally or alternatively, one or more parameters of a network security system may be adjusted based at least in part on the output document.

Claims

1. A method comprising:

configuring one or more web crawlers to obtain textual information from a plurality of data sources accessible over at least one network;

extracting terms likely to be associated with indicators of compromise from the obtained textual information;

filtering the extracted terms to identify terms corresponding to respective valid indicators of compromise;

generating links between the terms corresponding to the respective valid indicators of compromise;

converting the links and the corresponding terms into an output document in a specified indicator of compromise format;

transmitting the output document to an analyst device;

receiving feedback from the analyst device relating to the output document; and

adjusting at least one filter parameter of the filtering based at least in part on the received feedback;

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the data sources comprise public sources of textual descriptions of computer security threats.

3. The method of claim 1 wherein the data sources comprise one or more of forums, blogs and social media web sites.

4. The method of claim 1 wherein configuring one or more web crawlers further comprises:

configuring a web crawler of a first type to obtain first portions of the textual information from one or more of the data sources that render content statically without execution of client-side code; and

configuring a web crawler of a second type to obtain second portions of the textual information from one or more of the data sources that render content dynamically through execution of client-side code.

5. The method of claim 1 wherein configuring one or more web crawlers further comprises configuring a given web crawler to download portions of the textual information from a search application programming interface of a social media website.

6. The method of claim 1 wherein configuring one or more web crawlers further comprises seeding a given one of the web crawlers with a list of web sites identified in one or more messages received from an analyst device.

7. The method of claim 1 further comprising parsing the obtained textual information to identify portions of the obtained textual information that include the terms likely to be associated with the indicators of compromise.

8. The method of claim 1 wherein extracting terms likely to be associated with indicators of compromise from the obtained textual information further comprises identifying terms that exhibit one or more predefined indicator of compromise characteristics.

9. The method of claim 8 wherein the predefined indicator of compromise characteristics comprise one or more of Internet Protocol address characteristics and message digest characteristics.

10. The method of claim 1 wherein filtering the extracted terms to identify terms corresponding to respective valid indicators of compromise further comprises:

characterizing network domains from which respective ones of the extracted terms originated; and

filtering the extracted terms based at least in part on the characterized network domains.

11. The method of claim 1 wherein generating links between the terms corresponding to the respective valid indicators of compromise further comprises:

identifying semantic meanings of respective portions of the textual information that each include two or more of the valid indicators of compromise; and

generating at least one link between the two or more valid indicators of compromise in a given one of the portions of the textual information based at least in part on the identified semantic meaning of that portion.

12. The method of claim 1 further comprising adjusting one or more parameters of a network security system based at least in part on the output document.

13. A method comprising:

configuring one or more web crawlers to obtain textual information from a plurality of data sources accessible over at least one network;

extracting terms likely to be associated with indicators of compromise from the obtained textual information;

filtering the extracted terms to identify terms corresponding to respective valid indicators of compromise;

generating links between the terms corresponding to the respective valid indicators of compromise; and

converting the links and the corresponding terms into an output document in a specified indicator of compromise format;

the method further comprising;

parsing the obtained textual information to identify portions of the obtained textual information that include the terms likely to be associated with the indicators of compromise;

wherein parsing the obtained textual information further comprises;

parsing HTML code of each of one or more web pages into a plurality of different regions;

determining a quantity of text in each of the different regions; and

selecting one or more of the regions to be subject to the extracting based at least in part on the determined quantities of text;

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

14. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to configure one or more web crawlers to obtain textual information from a plurality of data sources accessible over at least one network;

to extract terms likely to be associated with indicators of compromise from the obtained textual information;

to filter the extracted terms to identify terms corresponding to respective valid indicators of compromise;

to generate links between the terms corresponding to the respective valid indicators of compromise;

to convert the links and the corresponding terms into an output document in a specified indicator of compromise format;

to transmit the output document to an analyst device;

to receive feedback from the analyst device relating to the output document; and

to adjust at least one filter parameter of the filtering based at least in part on the received feedback.

15. The computer program product of claim 14 wherein extracting terms likely to be associated with indicators of compromise from the obtained textual information further comprises identifying terms that exhibit one or more predefined indicator of compromise characteristics.

16. The computer program product of claim 14 wherein generating links between the terms corresponding to the respective valid indicators of compromise further comprises:

identifying semantic meanings of respective portions of the textual information that each include two or more of the valid indicators of compromise; and

generating at least one link between the two or more valid indicators of compromise in a given one of the portions of the textual information based at least in part on the identified semantic meaning of that portion.

17. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

said at least one processing device being configured;

to direct one or more web crawlers to obtain textual information from a plurality of data sources accessible over at least one network;

to extract terms likely to be associated with indicators of compromise from the obtained textual information;

to filter the extracted terms to identify terms corresponding to respective valid indicators of compromise;

to generate links between the terms corresponding to the respective valid indicators of compromise;

to convert the links and the corresponding terms into an output document in a specified indicator of compromise format;

to transmit the output document to an analyst device;

to receive feedback from the analyst device relating to the output document; and

to adjust at least one filter parameter of the filtering based at least in part on the received feedback.

18. The apparatus of claim 17 wherein said at least one processing device is further configured to adjust one or more parameters of a network security system based at least in part on the output document.

19. The apparatus of claim 17 wherein filtering the extracted terms to identify terms corresponding to respective valid indicators of compromise further comprises:

characterizing network domains from which respective ones of the extracted terms originated; and

filtering the extracted terms based at least in part on the characterized network domains.

20. The apparatus of claim 17 wherein generating links between the terms corresponding to the respective valid indicators of compromise further comprises:

identifying semantic meanings of respective portions of the textual information that each include two or more of the valid indicators of compromise; and

generating at least one link between the two or more valid indicators of compromise in a given one of the portions of the textual information based at least in part on the identified semantic meaning of that portion.