AUTOMATIC BOTNET SPAM SIGNATURE GENERATION

• US 20090265786A1
• Filed: 04/17/2008
• Published: 10/22/2009
• Est. Priority Date: 04/17/2008
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Osipkov, Ivan, Hulten, Geoffrey J., Achan, Kannan, Panigrahy, Rina, Yu, Fang, Xie, Yinglian
• Application Number:
  US12/104,441
• Publication Number:
  US 20090265786A1
• Time in Patent Office: Days
• US Class Current: 726/24
• CPC Class Codes: G06F 21/564 by virus signature recognition
• CPC Class Codes: G06F 2221/2145 Inheriting rights or properties, e.g., propagation of permissions or restrictions within a hierarchy
• CPC Class Codes: H04L 2463/144 Detection or countermeasures against botnets
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 63/126 the source of the received data
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

A framework may be used for generating URL signatures to identify botnet spam and membership. The framework may take a set of unlabeled emails as input that are grouped based on URLs contained within the emails. The framework may return a set of spam URL signatures and a list of corresponding botnet host IP addresses by analyzing the URLs within the emails that are contained within the groups. Each URL signature may be in the form of either a complete URL string or a URL regular expression. The signatures may be used to identify spam emails launched from botnets, while the knowledge of botnet host identities can help filter other spam emails also sent by them.

Claims

1. A system for generating uniform resource locator (URL) signatures to identify botnet spam and membership, comprising:

a URL preprocessor that extracts a plurality of URLs from a plurality of input emails and groups the input emails into a plurality of URL groups according to their corresponding domains;

a group selector that selects the URL groups in accordance with a predetermined feature; and

a regular expression generator that determines a signature representative of the URLs contained within a botnet spam.

2. The system of claim 1, wherein the predetermined feature is one of a sending time burstiness, a distribution of an internet protocol (IP) address space, or a specificity of the signature.

3. The system of claim 2, wherein for each URL, the group selector selects a group of URLs that exhibit the strongest temporal correlation across a set of distributed senders.

4. The system of claim 3, wherein a discrete time signal, reflecting a number of distinct source IP addresses that were active during a time window, is determined to represent the temporal correlation among distributed senders.

5. The system of claim 2, wherein for each determined signature, an entropy reduction based metric is used to quantify a specificity of the signature.

6. The system of claim 2, wherein the distribution is quantified using the total number of autonomous systems spanned by source IP addresses within the IP address space.

7. The system of claim 1, wherein the group selector associates an email with multiple groups if the email contains multiple URLs from different domains.

8. The system of claim 1, wherein the signature comprises one of a complete URL based signature or a regular expression based signature for a set of URLs belonging to a same domain.

9. The system of claim 8, wherein emails that match the complete URL based signature or regular expression based signature are identified as botnet sent spam emails.

10. The system of claim 9, wherein IP addresses corresponding to senders of the botnet sent spam emails are identified, and wherein each signature distinguishes a unique group of botnet hosts under the control of a common command and control computer.

11. The system of claim 10, wherein the complete URL based signature or regular expression based signature and the IP addresses are used to filter future spam emails.

12. A computer-implemented method for generating uniform resource locator (URL) signatures to identify botnet spam and membership, comprising:

extracting a plurality of URLs from a plurality of received emails;

grouping the emails into a plurality of groups according to a domain specified by the extracted URLs;

selecting the groups in accordance with a sending time burstiness or a distribution of an internet protocol (IP) address space of the emails within the groups; and

generating a signature representative of URLs contained within a botnet spam in accordance with the sending time burstiness or distribution of the IP address space to identify emails as being botnet spam.

13. The computer-implemented method of claim 12, further comprising:

selecting a group that exhibits a strongest temporal correlation across a set of distributed senders;

determining a signal spike within the group indicative of a number of IP addresses sending URLs targeting a common domain within a predetermined duration; and

ranking the group based on the signal spike.

14. The computer-implemented method of claim 12, further comprising:

quantifying the distribution using a total number of autonomous systems spanned by source IP addresses within the IP address space.

15. The computer-implemented method of claim 12, further comprising:

generating complete URL based signatures or regular expression based signatures for a set of URLs belonging to a same domain.

16. The computer-implemented method of claim 15, further comprising:

applying the complete URL based signature to detect spam emails that contain an identical URL string to the complete URL based signature; and

applying the regular expression based signatures to detect spam emails that contain polymorphic URLs.

17. The computer-implemented method of claim 15, further comprising:

receiving a set of polymorphic URLs from a same domain; and

constructing a keyword based signature tree to generate the regular expression based signatures.

18. A computer-implemented method for generating a spam signature to identify botnet spam and membership, comprising:

grouping a plurality of emails into a plurality of groups according to a domain specified by a plurality of uniform resource locators (URLs) within the emails;

iteratively selecting the groups in accordance with a sending time burstiness or a distribution of an internet protocol (IP) address space of the emails within the groups;

generating URL based signatures or regular expression based signatures for a set of URLs belonging to a same domain; and

outputting the URL based signature and a regular expression based signature to a spam filter.

19. The computer-implemented method of claim 18, further comprising:

applying the URL based signature to detect spam emails that contain an identical URL string to the complete URL based signature; and

applying the regular expression based signatures to detect spam emails that contain polymorphic URLs.

20. The computer-implemented method of claim 18, further comprising:

generating regular expressions from different domains and similar structures into a domain-agnostic regular expression; and

applying the regular expressions to capture spam emails that include URLs having different domains and a same URL structure.