SYSTEM AND METHODS FOR DETECTING SOFTWARE VULNERABILITIES AND MALICIOUS CODE

• US 20090165135A1
• Filed: 12/18/2008
• Published: 06/25/2009
• Est. Priority Date: 12/20/2007
• Status: Active Grant

• Current Assignee: Cybernet Systems Corporation
• Original Assignee: Cybernet Systems Corporation
• Inventors: Lomont, Chris C., Jacobus, Charles J.
• Granted Patent: US 8,806,619 B2
• Time in Patent Office: Days
• US Class Current: 726/22
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities

Abstract

A system and method determines whether software includes malicious code. A validation machine is instrumented with tools and monitors that capture the static and dynamic behavior of software. Software under examination is executed on the validation machine, and the tools and monitors are used to log data representative of the behavior of the software to detect vulnerable or malicious code. If possible, one or more operations are automatically performed on the software to enhance the security of the software by neutralizing the vulnerable or malicious code. Activities that cannot be neutralized automatically are flagged for human inspection. The software executed on the validation machine may be source code or non-source code, with different operations being disclosed and described in each case.

Claims

1. A method of determining whether software includes malicious code, comprising the steps of:

providing a validation machine;

instrumenting the validation machine with tools and monitors that capture the static and dynamic behavior of software;

executing software on the validation machine;

using the tools and monitors to log data representative of the behavior of the software during execution to detect vulnerable or malicious code;

automatically performing one or more operations on the software to enhance the security of the software by neutralizing the vulnerable or malicious code; and

flagging for human inspection activities that cannot be neutralized automatically.

2. The method of claim 1, including the step of training the validation machine to automatically neutralize vulnerable or malicious code using knowledge gained through the human inspection.

3. The method of claim 1, including the step of installing drivers to monitor resources of interest by hooking into system calls.

4. The method of claim 1, including the step of logging data representative of mouse, keyboard or other user inputs to determine if the software is performing suspicious operations with no associated user input.

5. The method of claim 1, including the step of using a firewall to monitor network access.

6. The method of claim 1, including the step of using internal operating system instrumentation to monitor and log activities of interest.

7. The method of claim 1, including the steps of:

storing a library of known trusted programs or malicious programs; and

comparing the software to the stored programs to detect vulnerable or malicious code.

8. The method of claim 1, including the step of using data mining techniques to find patterns indicative of malicious code.

9. The method of claim 1, including the steps of:

interfacing the machine to an unprotected network; and

monitoring hacker activities.

10. The method of claim 1, including the step of training a Bayesian network to recognize patterns between the logs and classify the behavior as likely safe or likely unsafe.

11. The method of claim 1, wherein the software executed on the validation machine is source code.

12. The method of claim 11, wherein the tools and monitors include source-code analysis tools to detect buffer overflows.

13. The method of claim 11, wherein the step of performing one or more operations on the software includes analyzing the source code to determine whether the source code is poorly written.

14. The method of claim 11, wherein the step of performing one or more operations on the software includes running the source code through a software tool operative to spot security errors.

15. The method of claim 11, wherein the step of performing one or more operations on the software includes linking libraries (libs) to the software to enhance security.

16. The method of claim 11, wherein the step of performing one or more operations on the software includes recompiling the code with a compiler that has stack checking routines.

17. The method of claim 11, wherein the step of performing one or more operations on the software includes one or more buffer checks.

18. The method of claim 11, wherein the step of performing one or more operations on the software includes automatically generating tests of small functions to detect weak points.

19. The method of claim 1, wherein source code is not available for the software executed on the validation machine.

20. The method of claim 19, wherein the tools and monitors include dynamic disassembly tools to determine where executable code was in memory, and then tag it as trusted or untrusted as it executes.

21. The method of claim 19, wherein the step of performing one or more operations on the software includes logging file, registry, configuration changes, network access, and other resource usage.

22. The method of claim 19, wherein the step of performing one or more operations on the software includes running the code in low-privilege modes and logging access to items not allowed to the program.

23. The method of claim 9, wherein the step of performing one or more operations on the software includes virus detection.

24. The method of claim 9, wherein the step of performing one or more operations on the software includes rootkit detection.

25. The method of claim 9, wherein the step of performing one or more operations on the software includes feeding bad data to auto hack programs.

26. The method of claim 9, wherein the step of performing one or more operations on the software includes monitoring network traffic and log activity with inbound or outbound firewalls.

27. The method of claim 1, wherein the validation machine is a virtual machine.

28. The method of claim 1, further including the step of rating the software as safe or unsafe with some probability.