Systems and methods for detecting encrypted bot command and control communication channels

• US 8,204,984 B1
• Filed: 11/30/2007
• Issued: 06/19/2012
• Est. Priority Date: 04/01/2004
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Aziz, Ashar, Lai, Wei-Lung, Manni, Jayaraman
• Primary Examiner(s): LAZARO, DAVID R
• Application Number:
  US11/998,750
• Time in Patent Office: 1,663 Days
• Field of Search: 709/223, 709/224, 726 22- 25
• US Class Current: 709/224
• CPC Class Codes: H04L 2463/144 Detection or countermeasures against botnets
• CPC Class Codes: H04L 63/0227 Filtering policies mail message filtering H04L51/212
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

Methods and systems for detecting encrypted bot command and control communication channels are provided. In the exemplary method, the presence of a communication channel between a first network device and a second network device is monitored. Active and inactive periods of the network device are detected and a reverse channel is determined based on the detection. The first network device may then be flagged as potentially infected or suspected based on the reverse channel determination.

Claims

1. A system comprising:

a channel monitoring module configured to monitor a channel between a first network device and a second network device;

an active/inactive detector module configured to detect an active period and an inactive period of the first network device;

a reverse channel detection module comprising instructions stored on a computer readable medium, the reverse channel detection module being configured to detect an establishment of communication by the first network device and a direction for communication over the channel thereafter based on the detected active period and the inactive period of the first network device with respect to the second network device, the reverse channel detection module being further configured to determine the reverse channel based on communications over the channel being substantially in a reverse direction after the detected establishment; and

a flagging module configured to flag the first network device as potentially infected by a bot based on the reverse channel determination.

2. The system of claim 1, further comprising a whitelist module for determining if the first network device is associated with a white list.

3. The system of claim 1, wherein the channel monitoring module is further configured to determine if an IRC channel is established by the first network device.

4. The system of claim 1, further comprising a network scanning module configured to determine if the first network device scans a network.

5. The system of claim 1, further comprising a controller configured to simulate data flow between the first network device and the second network device.

6. The system of claim 5, wherein the controller comprises a replayer and a virtual machine, the replayer configured to transmit the data flow to the virtual machine.

7. The system of claim 6, wherein an infection by the bot is confirmed based on an analysis of a response of the virtual machine.

8. The system of claim 5, wherein the controller further comprises a signature module configured to generate a signature to identify the bot.

9. The system of claim 8, wherein the controller is further configured to send the signature to a bot detector.

10. The system of claim 1, wherein the flagging module is further configured to assign a color category to the first network device for review by an administrator.

11. A method comprising:

monitoring a channel between a first network device and a second network device;

detecting an active period and an inactive period of the first network device;

determining a reverse channel based on the detected active period and the inactive period of the first network device with respect to the second network device, the determining including detecting establishment of communication by the first network and a direction for communication over the channel thereafter, and detecting communications over the channel being substantially in a reverse direction after the detected establishment; and

flagging the first network device as potentially infected by a bot based on the reverse channel determination.

12. The method of claim 11, further comprising determining if the reverse channel is associated with a white list.

13. The method of claim 11, further comprising determining if an IRC channel is established by the first network device.

14. The method of claim 11, further comprising determining if the first network device scans a network.

15. The method of claim 11, further comprising simulating a data flow between the first network device and the second network device.

16. The method of claim 15, further comprising transmitting the data flow to a virtual machine.

17. The method of claim 15, further comprising confirming an infection by the bot based on analysis of a response of the virtual machine.

18. The method of claim 11, further comprising generating a signature to identify the bot.

19. The method of claim 18, further comprising sending the signature to a bot detector.

20. A non-transitory computer readable medium having embodied thereon executable instructions, the instructions being executable by a processor for detecting encrypted bot command &

control communication channels, the method comprising;

monitoring a channel between a first network device and a second network device;

detecting an active period and an inactive period of the first network device;

determining a reverse channel based on the detected active period and the inactive period of the first network device with respect to the second network device, the determining including detecting establishment of communication by the first network and a direction for communication over the channel thereafter, and detecting communications over the channel being substantially in a reverse direction after the detected establishment; and

flagging the first network device as potentially infected by a bot based on the reverse channel determination.