Technique for detecting executable malicious code using a combination of static and dynamic analyses

• US 20050108562A1
• Filed: 06/18/2003
• Published: 05/19/2005
• Est. Priority Date: 06/18/2003
• Status: Abandoned Application

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Rabek, Jesse C., Khazan, Roger I., Cunningham, Robert K., Lewandowski, Scott M.
• Application Number:
  US10/464,828
• Publication Number:
  US 20050108562A1
• Time in Patent Office: Days
• US Class Current: 726/23
• CPC Class Codes: G06F 11/3604 Software analysis for verifying properties of programs byte-code verification G06F9/44589
• CPC Class Codes: G06F 21/563 by source code analysis
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities

Abstract

Described are techniques used for automatic detection of malicious code by verifying that an application executes in accordance with a model defined using calls to a predetermined set of targets, such as external routines. A model is constructed using a static analysis of a binary form of the application, and is comprised of a list of calls to targets, their invocation and target locations, and possibly other call-related information. When the application is executed, dynamic analysis is used to intercept calls to targets and verify them against the model. The verification may involve comparing the invocation and target location, as well as other call-related information, available at the time of call interception to the corresponding information identified by static analysis. A failed verification determines that the application includes malicious code. As an option, once detected, the malicious code may be allowed to execute to gather information about its behavior.

Claims

1. A method for detecting malicious code comprising:

performing static analysis of an application prior to execution of the application identifying any invocations of at least one predetermined target routine;

determining, prior to executing said at least one predetermined target routine during execution of the application, whether a run time invocation of the at least one predetermined target routine has been identified by said static analysis as being invoked from a predetermined location in said application; and

if the run time invocation of the at least one predetermined target routine has not been identified from a predetermined location by said static analysis, determining that the application includes malicious code.

2. The method of claim 1, wherein said target routines are external to the said application, and the method further comprising:

instrumenting a binary form of a library such that all invocations of a predetermined set of one or more external routines included in said library are intercepted; and

intercepting an invocation instance of one of said external routines.

3. The method of claim 2, wherein said predetermined set includes a portion of system functions.

4. The method of claim 1, further comprising, performing, prior to executing said application:

determining an invocation location within said application from which a call to said at least one predetermined target routine is made; and

determining a target location associated with said invocation location, said target location corresponding to said at least one predetermined target routine to which said call is being made.

5. The method of claim 4, further comprising:

using said invocation location and said associated target location to determine, during execution of said application, whether an intercepted call to target routine has been identified by said static analysis.

6. The method of claim 1, wherein static analysis is performed after beginning execution of instructions of said application and prior to execution of said at least one predetermined routine.

7. The method of claim 1, wherein said static analysis is performed as a preprocessing step prior to at least one of:

invoking said application and executing a first instruction of said application.

8. The method of claim 7, wherein a single static analysis is performed producing static analysis results used in connection with a plurality of subsequent executions of said application.

9. The method of claim 1, wherein said target routines are external to the said application, further comprising:

loading said application into memory such that said application is in a suspended execution state;

instrumenting a library used by said application such that invocations of a predetermined set of one or more external routines defined in said library are intercepted; and

executing said application and intercepting an invocation instance of one of said predetermined set of external routines.

10. The method of claim 1 further comprising:

prior to executing a target routine, intercepting a call to the target routine and verifying that the call to the target routine has been identified by said static analysis.

11. The method of claim 10, further comprising:

intercepting control after completing execution of said target routine.

12. The method of claim 10, further comprising:

determining that the application includes malicious code.

13. The method of claim 12, further comprising:

halting execution of said application upon determining that said application includes malicious code.

14. The method of claim 11, further comprising:

determining that the application includes malicious code;

determining if malicious code analysis is being performed;

if malicious code analysis is being performed;

executing the intercepted target routine that failed verification; and

intercepting control after said target routine has completed executing.

15. The method of claim 5, wherein said static analysis includes obtaining parameter information associated with target routine calls made from said application, and the method comprising:

using said parameter information in verifying an intercepted target routine call by comparing said parameter information of said static analysis to other parameter information of said intercepted target routine call wherein said verifying is used in determining whether said application includes malicious code.

16. The method of claim 1, further comprising:

determining which target routines are executed by one or more other applications; and

intercepting any invocations of said target routines during execution of said application.

17. The method of claim 9, wherein further comprising, performing after said loading prior to executing said application:

copying, for each of said one or more external routines, a first set of one or more instructions to a save area corresponding to said each external routine; and

modifying a memory copy of each of said one or more external routines to transfer control to a wrapper routine corresponding to said each external routine prior to executing any instructions of said each external routine and after all instructions of said each external routine have been executed.

18. The method of claim 10, further comprising:

obtaining run time context information about said target routine including at least one of;

run time call chain information before executing an intercepted target routine, run time call chain information after executing an intercepted target routine, run time stack information, and return parameter information.

19. The method of claim 12, further comprising, if malicious code analysis is not being performed, performing at least one of the following:

returning a return value corresponding to a predetermined error code, and transferring run time control to routine-specific condition handler.

20. The method of claim 2, wherein said instrumentation is performed as a preprocessing step prior to executing an instruction of said application.

21. The method of claim 2, wherein said instrumentation is performed dynamically after beginning execution of instructions of said application.

22. The method of claim 2, wherein said instrumenting produces an instrumented version of said binary form and said instrumented version is stored on a storage device.

23. The method of claim 2, wherein said instrumenting produces an instrumented version of said binary form and said instrumented version is stored in memory.

24. The method of claim 5, further comprising, for determining said invocation location:

obtaining a return address from a run time stack, said return address identifying an address of said application to which control is returned after completion of a call to a target routine;

using said return address to determine another location in said application of a previous instruction immediately prior to said return address;

determining whether contents of said other location includes an expected transfer instruction; and

if said contents does not include an expected transfer instruction, determining that said application includes malicious code.

25. The method of claim 24, further comprising:

determining, using information obtained from static analysis of said application, whether other locations of said application include at least one element of expected information associated with the expected transfer instruction; and

if said contents does not include said at least one element of expected information, determining that said application includes malicious code.

26. The method of claim 24, wherein said information of said other location is obtained from a copy of said application stored on a storage device.

27. The method of claim 24, wherein said information of said other location is obtained from a copy of said application stored in a memory.

28. The method of claim 1, wherein said application is one of:

bytecode form, script language form, and command language form, and the method further comprises;

instrumenting one of;

a processor of said application and said application such that all invocations of a predetermined set of one or more target routines invoked by said application are intercepted; and

intercepting an invocation instance of one of said target routines.

29. The method of claim 14, further comprising, if malicious code analysis is being performed:

obtaining run time context information about said malicious code including at least one of;

return parameter information, run time call chain information before executing an intercepted target routine, run-time chain after executing an intercepted target routine and run time stack information.

30. The method of claim 7, wherein the results of said static analysis are stored on at least one of:

a storage device and a memory.

31. The method of claim 1, wherein the said static analysis is performed on a host on which the application is executed.

32. The method of claim 1, wherein said static analysis is performed on a first host and static analysis results are made available to a second host on which said application is executed.

33. The method of claim 1, wherein the results of said static analysis are distributed together with the said application.

34. The method of claim 1, wherein said target routines are external to the said application, and the method further comprising:

using an instrumented version of a binary form of a library such that all invocations of a predetermined set of one or more external routines included in said library are intercepted; and

intercepting an invocation instance of one of said external routines.

35. The method of claim 34, wherein said instrumented version of said binary form obtained from at least one of:

a data storage system and a host other than a host on which said application is executed, and said instrumented version is stored on a storage device.

36. A method for detecting malicious code comprising:

determining, prior to executing at least one predetermined target routine during execution of the application, whether a run time invocation of the at least one predetermined target routine is identified by a model as being invoked from a predetermined location in said application, said model identifying locations within said application from which invocations of the at least one predetermined target routine occur; and

if the run time invocation of the at least one predetermined target routine has not been identified from a predetermined location by said model, determining that the application includes malicious code.

37. A method for detecting malicious code comprising:

obtaining static analysis information of an application identifying any invocations of at least one predetermined target routine;

determining, prior to executing said at least one predetermined target routine during execution of the application, whether a run time invocation of the at least one predetermined target routine has been identified by said static analysis information as being invoked from a predetermined location in said application; and

if the run time invocation of the at least one predetermined target routine has not been identified from a predetermined location by said static analysis information, determining that the application includes malicious code.

38. A computer program product that detects malicious code comprising:

executable code that performs static analysis of an application prior to execution of the application identifying any invocations of at least one predetermined target routine;

executable code that determines, prior to executing said at least one predetermined target routine during execution of the application, whether a run time invocation of the at least one predetermined target routine has been identified by said static analysis as being invoked from a predetermined location in said application; and

executable code that, if the run time invocation of the at least one predetermined target routine has not been identified from a predetermined location by said static analysis, determines that the application includes malicious code.

39. The computer program product of claim 38, wherein said target routines are external to the said application, the computer program product further comprising:

executable code that instruments a binary form of a library such that all invocations of a predetermined set of one or more external routines included in said library are intercepted; and

executable code that intercepts an invocation instance of one of said external routines.

40. The computer program product of claim 39, wherein said predetermined set includes a portion of system functions.

41. The computer program product of claim 38, further comprising, executable code that, prior to executing said application:

determines an invocation location within said application from which a call to said at least one predetermined target routine is made; and

determines a target location associated with said invocation location, said target location corresponding to said at least one predetermined target routine to which said call is being made.

42. The computer program product of claim 41, further comprising:

executable code that uses said invocation location and said associated target location to determine, during execution of said application, whether an intercepted call to target routine has been identified by said static analysis.

43. The computer program product of claim 38, wherein static analysis is performed after beginning execution of instructions of said application and prior to execution of said at least one predetermined routine.

44. The computer program product of claim 38, wherein said static analysis is performed as a preprocessing step prior to at least one of:

invoking said application and executing a first instruction of said application.

45. The computer program product of claim 44, wherein a single static analysis is performed producing static analysis results used in connection with a plurality of subsequent executions of said application.

46. The computer program product of claim 38, wherein said target routines are external to the said application, the computer program product further comprising:

executable code that loads said application into memory such that said application is in a suspended execution state;

executable code that instruments a library used by said application such that invocations of a predetermined set of one or more external routines defined in said library are intercepted; and

executable code that executes said application and intercepting an invocation instance of one of said predetermined set of external routines.

47. The computer program product of claim 38, further comprising:

executable code that, prior to executing a target routine, intercepts a call to the target routine and verifies that the call to the target routine has been identified by said static analysis.

48. The computer program product of claim 47, further comprising:

executable code that intercepts control after completing execution of said target routine.

49. The computer program product of claim 47, further comprising:

executable code that determines that the application includes malicious code.

50. The computer program product of claim 49, further comprising:

executable code that halts execution of said application upon determining that said application includes malicious code.

51. The computer program product of claim 48, further comprising:

executable code that determines that the application includes malicious code;

executable code that determines if malicious code analysis is being performed;

executable code that, if malicious code analysis is being performed;

executes the intercepted target routine that failed verification; and

intercepts control after said target routine has completed executing.

52. The computer program product of claim 42, wherein said static analysis includes obtaining parameter information associated with target routine calls made from said application, and the computer program product comprising:

executable code that uses said parameter information in verifying an intercepted target routine call by comparing said parameter information of said static analysis to other parameter information of said intercepted target routine call wherein said verifying is used in determining whether said application includes malicious code.

53. The computer program product of claim 38, further comprising:

executable code that determines which target routines are executed by one or more other applications; and

executable code that intercepts any invocations of said target routines during execution of said application.

54. The computer program product of claim 46, further comprising executable code that, after said loading prior to executing said application:

copies, for each of said one or more external routines, a first set of one or more instructions to a save area corresponding to said each external routine; and

modifies a memory copy of each of said one or more external routines to transfer control to a wrapper routine corresponding to said each external routine prior to executing any instructions of said each external routine and after all instructions of said each external routine have been executed.

55. The computer program product of claim 47, further comprising:

executable code that obtains run time context information about said target routine including at least one of;

run time call chain information before executing an intercepted target routine, run time call chain information after executing an intercepted target routine, run time stack information, and return parameter information.

56. The computer program product of claim 49, further comprising, if malicious code analysis is not being performed, performing at least one of the following:

returning a return value corresponding to a predetermined error code, and transferring run time control to routine-specific condition handler.

57. The computer program product of claim 39, wherein said executable code that instruments is executed as a preprocessing step prior to executing an instruction of said application.

58. The computer program product of claim 39, wherein said executable code that instruments is executed so that instrumentation is dynamically performed after beginning execution of instructions of said application.

59. The computer program product of claim 39, wherein said executable code that instruments produces an instrumented version of said binary form and said instrumented version is stored on a storage device.

60. The computer program product of claim 39, wherein said executable code that instruments produces an instrumented version of said binary form and said instrumented version is stored in memory.

61. The computer program product of claim 42, wherein said executable code that determines said invocation location, further comprises executable code that:

obtains a return address from a run time stack, said return address identifying an address of said application to which control is returned after completion of a call to a target routine;

uses said return address to determine another location in said application of a previous instruction immediately prior to said return address;

determines whether contents of said other location includes an expected transfer instruction; and

if said contents does not include an expected transfer instruction, determines that said application includes malicious code.

62. The computer program product of claim 61, further comprising:

executable code that determines, using information obtained from static analysis of said application, whether other locations of said application include at least one element of expected information associated with the expected transfer instruction; and

executable code that, if said contents does not include said at least one element of expected information, determines that said application includes malicious code.

63. The computer program product of claim 61, wherein said information of said other location is obtained from a copy of said application stored on a storage device.

64. The computer program product of claim 61, wherein said information of said other location is obtained from a copy of said application stored in a memory.

65. The computer program product of claim 38, wherein said application is one of:

bytecode form, script language form, or command language form, and the computer program product further comprises;

executable code that instruments one of;

a processor of said application and said application such that all invocations of a predetermined set of one or more target routines invoked by said application are intercepted; and

executable code that intercepts an invocation instance of one of said target routines.

66. The computer program product of claim 51, further comprising, executable code that, if malicious code analysis is being performed:

obtains run time context information about said malicious code including at least one of;

return parameter information, run time call chain information before executing an intercepted target routine, run-time chain after executing an intercepted target routine and run time stack information.

67. The computer program product of claim 44, further comprising executable code that stores the results of static analysis on at least one of:

a storage device and a memory.

68. The computer program product of claim 38, wherein the said executable code that performs static analysis is executed on a host on which the application is executed.

69. The computer program product of claim 38, wherein said executable code that performs static analysis is executed on a first host and the computer program product further includes executable code that makes static analysis results available to a second host on which said application is executed.

70. The computer program product of claim 38, wherein the results of said static analysis are distributed together with the said application.

71. The computer program product of claim 38, wherein said target routines are external to the said application, and the computer program product further comprising:

executable code that uses an instrumented version of a binary form of a library such that all invocations of a predetermined set of one or more external routines included in said library are intercepted; and

executable code that intercepts an invocation instance of one of said external routines.

72. The computer program product of claim 71, wherein said instrumented version of said binary form obtained from at least one of:

a data storage system and a host other than a host on which said application is executed, and said instrumented version is stored on a storage device.

73. A computer program product that detects malicious code comprising:

executable code that determines, prior to executing at least one predetermined target routine during execution of the application, whether a run time invocation of the at least one predetermined target routine is identified by a model as being invoked from a predetermined location in said application, said model identifying locations within said application from which invocations of the at least one predetermined target routine occur; and

executable code that, if the run time invocation of the at least one predetermined target routine has not been identified from a predetermined location by said model, determines that the application includes malicious code.

74. A computer program product that detects malicious code comprising:

executable code that obtains static analysis information of an application identifying any invocations of at least one predetermined target routine;

executable code that determines, prior to executing said at least one predetermined target routine during execution of the application, whether a run time invocation of the at least one predetermined target routine has been identified by said static analysis information as being invoked from a predetermined location in said application; and

executable code that, if the run time invocation of the at least one predetermined target routine has not been identified from a predetermined location by said static analysis information, determines that the application includes malicious code.