PROBABILISTIC SHELLCODE DETECTION

• US 20100031359A1
• Filed: 04/15/2008
• Published: 02/04/2010
• Est. Priority Date: 04/14/2008
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: Secure Computing Corporation (McAfee, LLC)
• Inventors: Alme, Christoph
• Granted Patent: US 8,549,624 B2
• Time in Patent Office: Days
• US Class Current: 726/24
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements

Abstract

Various embodiments include a method of detecting shell code in an arbitrary file comprising determining where one or more candidate areas exist within an arbitrary file, searching at least one nearby area surrounding each of the one or more candidate areas within the arbitrary file for an instruction candidate, and calculating for any such instruction candidate a statistical probability based on a disassembly of instructions starting at a found offset for the instruction candidate that the disassembled instructions are shellcode.

Claims

1. A method of detecting shell code in an arbitrary file comprising:

determining where one or more candidate areas exist within an arbitrary file;

searching at least one nearby area surrounding each of the one or more candidate areas within the arbitrary file for an instruction candidate; and

calculating for any such instruction candidate a statistical probability based on a disassembly of instructions starting at a found offset for the instruction candidate that the disassembled instructions are shellcode.

2. The method of claim 1, including:

classifying the arbitrary file as containing shellcode if the statistical probability exceeds a probability threshold.

3. The method of claim 1, wherein the instruction candidate at the found offset is a code branching instruction or a function call.

4. The method of claim 1, wherein the instruction candidate at the found offset is similar to the start of a known characteristical shellcode sequence.

5. The method of claim 1, wherein determining where one or more candidate areas exist within the arbitrary file includes:

scanning the arbitrary file looking for repetitive constructs, the repetitive construct potentially intended to overflow a buffer in a computer memory when the arbitrary file is parsed, rendered or executed.

6. The method of claim 1, wherein searching at least one nearby area surrounding each of the one or more candidate areas within the arbitrary file includes:

configuring a number of data blocks to be included in the at least one nearby area.

7. The method of claim 1, wherein calculating for any found offset a statistical probability based on a disassembly of instructions starting at the found offset includes:

disassembling the instructions starting at the any found offset by following branches and method calls dynamically.

8. The method of claim 1, wherein calculating for any found offset a statistical probability based on a disassembly of instructions starting at the found offset includes:

for a given stream of instructions starting at a given one of the any found offset, mapping an instruction-to-shellcode probability to each of the instruction in the given stream of instructions; and

summing the mapped instruction-to-shellcode probability for each instruction using Bayes'"'"' formula to generate an overall probability.

9. The method of claim 8, including classifying the arbitrary file as containing shellcode if the overall probability exceeds a threshold value.

10. A gateway comprising:

an anti-malware engine operable to scan an arbitrary file and to determine if any candidate areas exist within the arbitrary file;

for any given candidate area located within the arbitrary file, search at least one nearby area surrounding the candidate area for any instruction candidates; and

for any such instruction candidates, calculate a statistical probability based on one or more disassembled instructions starting at a found offset of the instruction candidate to determine a likelihood that the arbitrary file includes shellcode.

11. The gateway of claim 10, further including a configurations coupled to the anti-malware engine, the configurations operable to store one or more threshold values used by the anti-malware engine in at least one shellcode detection process.

12. The gateway of claim 10, wherein the anti-malware engine is operable to output warning messages to an interface coupled to the anti-malware engine when the arbitrary file is determined to include shellcode.

13. The gateway of claim 10, wherein the gateway couples at least one protected device to the Internet.

14. The gateway of claim 10, wherein the anti-malware engine is operable to access known characteristical shellcode sequences stored in a database coupled to the anti-malware engine, and to scan the arbitrary file for the known characteristical shellcode sequences.

15. A method of detecting shellcode in an arbitrary file comprising:

scanning an arbitrary file to determine if any candidate areas exist within the arbitrary file;

for any candidate areas found in the arbitrary file, first searching the areas surrounding the any candidate areas to determine if any function calls or any code branching instructions exist in the areas surrounding the any candidate areas; and

if no function calls and no code branching instructions are found, searching the areas surrounding the any candidate areas for known characteristical shellcode sequences.

16. The method of claim 15, including if no known characteristical shellcode sequences are found, scanning the arbitrary file for data blocks with high information entropy compared to a threshold value.

17. The method of claim 15, wherein scanning the arbitrary file includes scanning the arbitrary file for repeated constructs that include long sequences of repeated characters.

18. The method of claim 17, wherein scanning for repeated constructs includes:

taking a first character a_N at position N in the arbitrary file;

searching for next occurrence of the first character, which is found at position M;

using a sub-sequence (a_N . . . , a<sub>M−

1</sub>) as a repetition pattern having a length L=M−

N);

comparing the repetition pattern to the sub-sequence (a_M, . . . a_M+L), and if equal, advance by L bytes and continue with this comparing until the comparison fails in order to determine a number of found repetitions; and

if the number of found repetitions is below a given threshold, determine that no match is found.

19. The method of claim 15, wherein if any function calls or code branching instructions are found in the areas surrounding the any candidate areas,

disassemble a number of instructions starting at the function call or at the code branching instruction,

calculate a probability for each of the instructions disassembled;

sum each of the calculated probabilities using Bayers'"'"' formula to generate an overall probability; and

using the overall probability, determine that the arbitrary file includes shellcode when the overall probability exceeds an threshold value.

20. A computer network comprising:

a gateway including an anti-malware engine operable to perform the following;

receiving an arbitrary file;

scanning the arbitrary file for repetitive constructs that are potentially intended to overflow a buffer;

determining if any function calls or any code branching instructions exist in the areas surrounding the repetitive constructs; and

generating a statistical probability representing the likelihood that the arbitrary file includes shellcode by performing a statistical analysis of the instructions starting at each found offset to generate an overall shellcode probability for the instructions starting at the each found offset.

21. The computer network of claim 20, further including a configurations coupled to the anti-malware engine, the configurations operable to store one or more probability threshold settings used by the anti-malware engine in determining whether a given statistical probability generated for the arbitrary file indicates that the arbitrary file includes shellcode.

22. The computer network of claim 20, further including a database coupled to the gateway, the database including memory operable to store one or more known shellcode sequences and to provide the one or more known shellcode sequences to the anti-malware engine for comparison to instructions in the arbitrary file.