Secure execution of a computer program

• US 7,594,111 B2
• Filed: 12/18/2003
• Issued: 09/22/2009
• Est. Priority Date: 12/19/2002
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Bruening, Derek L., Kiriansky, Vladimir L., Amarasinghe, Saman P.
• Primary Examiner(s): Barron, Jr.; Gilberto
• Assistant Examiner(s): Almeida; Devin
• Application Number:
  US10/740,063
• Publication Number:
  US 20040133777A1
• Time in Patent Office: 2,105 Days
• Field of Search: 713/164, 713/165, 713/166, 713/167, 726/1, 726/2, 726/26, 726/17
• US Class Current: 713/166
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 21/554 involving event detection and direct action

Abstract

Hijacking of an application is prevented by monitoring control flow transfers during program execution in order to enforce a security policy. At least three basic techniques are used. The first technique, Restricted Code Origins (RCO), can restrict execution privileges on the basis of the origins of instruction executed. This distinction can ensure that malicious code masquerading as data is never executed, thwarting a large class of security attacks. The second technique, Restricted Control Transfers (RCT), can restrict control transfers based on instruction type, source, and target. The third technique, Un-Circumventable Sandboxing (UCS), guarantees that sandboxing checks around any program operation will never be bypassed.

Claims

1. A method for securing a computing system having a processor, the method comprising:

analyzing control transfers of a program as the program executes on said computing system, said program comprising instructions, each of the instructions being machine code of the processor, and said control transfers each caused by execution of corresponding control transfer instructions, the control transfer instructions causing a break in sequential flow of program execution, thereby determining a next instruction of the program to execute, said analyzing of the control transfers being performed prior to the execution of the control transfer instructions; and

ensuring that each said control transfer complies with a security policy, said security policy comprising a set of rules against which each of the control transfers is separately evaluated during the analyzing, the ensuring comprising announcing or handling a security violation when the control transfer does not comply with the security policy.

2. A method according to claim 1, wherein:

said step of ensuring includes restricting execution privileges based on code origins by checking an origin of code of said program against said security policy to determine if said code should be given execute privileges, said checking of said origin comprises determining whether said code is;

a) from an original image on disk and unmodified, b) dynamically generated but unmodified since generation or c) modified.

3. A method according to claim 1, wherein:

the control transfers are of different types, and said security policy includes a set of rules that are separately definable for each type of control transfer; and

the different types include at least;

direct call, indirect call, return, conditional branch, direct jump, indirect jump and interrupt.

4. A method according to claim 1, wherein:

said step of ensuring includes restricting control transfers based on instruction type.

5. A method according to claim 1, wherein:

said step of ensuring includes restricting at least one of said control transfers based on a source of said at least one of said control transfer instructions, said restricting implements a restriction which is specific to a type of said at least one of said control transfers.

6. A method according to claim 1, wherein:

said step of ensuring includes restricting at least one of said control transfers based on a target of said at least one of said control transfers, said restricting implements a restriction which is specific to a type of said at least one of said control transfers.

7. A method according to claim 1, wherein said step of ensuring includes requiring a pre-defined prologue before execution of a sequence of at least one of said instructions of the program, and/or a pre-defined epilogue after execution of said sequence.

8. A method according to claim 1, wherein:

said step of ensuring includes restricting execution privileges based on code origins, preventing bypass of sandboxing checks placed around a program operation and restricting the control transfers based on at least any one of source, destination or instruction class.

9. A method according to claim 1, wherein said step of ensuring includes:

determining whether code of said program at a target of said control transfer has been modified; and

preventing execution of said code of said program which has been determined to be modified.

10. A method according to claim 1, wherein:

said step of ensuring includes ensuring that shared library code is entered only through declared entry points.

11. A method according to claim 1, wherein:

said step of ensuring includes verifying all branch instructions of said program comply with the security policy.

12. A method according to claim 1, wherein:

said control transfer instructions include a return instruction; and

said step of ensuring includes requiring that a destination of said return instruction is immediately preceded by a direct or indirect call instruction.

13. A method according to claim 1, wherein:

at least one of said instructions of said program is from a corresponding code page of an original application or library image on disk; and

said step of ensuring includes determining whether said corresponding code page has been modified during execution of the program and allowing execution of said at least one of said instructions when said said corresponding code page has not been modified during execution of the program.

14. A method according to claim 1, wherein:

said step of ensuring includes verifying targets of calls against a predefined list.

28. A method according to claim 1, wherein:

said step of ensuring includes ensuring that all control transfers for said program comply with said security policy.

29. A method according to claim 1, wherein said step of ensuring determines whether instructions of said program at a target of the control transfers are from code pages which are unmodified during execution of said program.

30. A method according to claim 29, wherein said code pages are determined to be unmodified by maintaining a list of unmodified code pages.

31. A method according to claim 1, wherein said step of ensuring further comprises determining whether said instructions at a target of the control transfers were present when said program is originally loaded, and have not been modified after being loaded.

32. A method according to claim 31, wherein said instructions of said program are determined to be unmodified after being loaded by maintaining a list of modified code pages.

33. A method according to claim 1, wherein said step of ensuring includes determining whether a matching rule of said security policy is found for a source address, destination address and instruction type associated with at least one of said control transfers, and applying the matching rule to the associated control transfer.

34. A method according to claim 33, wherein said security policy includes different rules for different control transfer types, including types of at least:

direct call, indirect call, return, conditional branch, direct jump and interrupt.

42. A method according to claim 1, wherein:

each instruction is a machine language primitive which is directly executed on the processor.

43. A method according to claim 1, wherein:

the program comprises modules, one of the control transfers is a direct or indirect jump from a source to a destination, and the plurality of rules includes a rule for the direct or indirect jump which comprises determining whether the source and the destination of the direct or indirect jump are within the same module of the program.

44. A method according to claim 1, wherein:

the program comprises modules, each module comprises a group of functions, one of the control transfers is a direct jump to a target, and the plurality of rules includes a rule for the direct jump which comprises determining whether the target of the direct jump is within the direct jump'"'"'s own function.

45. A method according to claim 1, wherein:

one of the control transfers is a return to a destination, and the plurality of rules includes a rule for the return which comprises determining whether the destination is present in a valid return address list.

46. A method according to claim 1, wherein:

one of the control transfers is a call, and the plurality of rules includes a rule for the call which comprises determining whether a destination of the call is a call entry point.

47. A method according to claim 1, wherein:

the program comprises modules, one of the control transfers is a call to a destination, and the plurality of rules includes a rule for the call which comprises determining whether the call and the destination are both within the same module of the program.

48. A method according to claim 1, wherein:

one of the control transfers is a call to a callee, the call is contained in a module of the program, and the plurality of rules includes a rule for the call which comprises determining whether the callee is imported by the module that contains the call instruction.

49. A method according to claim 1, wherein:

one of the control transfers is a call to a callee, the callee is contained in a module of the program, and the plurality of rules includes a rule for the call which comprises determining whether the callee is exported by its containing module.

50. A method according to claim 1, wherein:

one of the control transfers is a call from at least one source, and the plurality of rules includes a rule for the call which comprises determining whether the source of the call is in a library module.

51. A method according to claim 1, wherein:

the program comprises modules, one of the control transfers is a jump from a source to a destination, and the plurality of rules includes a rule for the jump which comprises determining whether the source and the destination of the jump are within the same module of the program;

one of the control transfers is a return to a destination, and the plurality of rules includes a rule for the return which comprises determining whether the destination is present in a valid return address list; and

one of the control transfers is a call, and the plurality of rules includes a rule for the call which comprises determining whether a destination of the call is a call entry point.

52. A method according to claim 1, wherein:

the control transfers are of different types, and said set of rules is separately definable for each type of control transfer, the different types include;

exceptions and kernel mediated control transfers.

15. A method for securing a computing system having a processor, the method comprising:

accessing a set of rules for transferring control of a program on said computing system;

analyzing control transfers of said program as the program executes on said computing system, said program comprising instructions, each of the instructions being a member of an instruction set of the processor, and said control transfers each caused by execution of corresponding control transfer instructions, the transfer instruction causing a break in sequential flow of program execution, thereby determining a next instruction of the program to execute, said analyzing of the control transfers being performed prior to the execution of the control transfer instruction; and

enforcing said rules for said control transfers for said program, said enforcing comprising;

evaluating, during the analyzing, each of the control transfers against the rules; and

announcing or handling a security violation instead of a target of the control transfer instruction when the control transfer instruction does not comply with the rules.

16. A method according to claim 15, wherein:

said step of enforcing includes restricting execution privileges based on code origins, including determining whether code of said program is;

a) from an original image on disk and unmodified, b) dynamically generated but unmodified since generation or c) modified.

17. A method according to claim 15, wherein said step of enforcing includes restricting the control transfers based on a source of the control transfer instructions.

18. A method according to claim 15, wherein said step of enforcing includes restricting the control transfers based on the target of the control transfer instructions.

19. A method according to claim 15, wherein said step of enforcing includes requiring a pre-defined prologue before execution of a sequence of at least one of said instructions of the program and/or a pre-defined epilogue after execution of said sequence.

20. A method according to claim 15, wherein:

said step of enforcing includes restricting execution privileges based on code origins, preventing bypass of sandboxing checks placed around a program operation and restricting the control transfers based on at least any one of source, destination or instruction class.

35. A method according to claim 15, wherein:

said program comprises modules and said rules restrict intra-module control transfers.

36. A method according to claim 15, wherein:

said rules allow a return instruction of said program to execute when a destination of said return instruction is immediately preceded by a direct or indirect call instruction that was previously executed.

37. A method according to claim 15, wherein:

said rules include different rules for handling a control transfer which is conditional, a control transfer which is unconditional and a control transfer which is indirect.

38. A method according to claim 15, wherein:

at least one of said control transfers comprises a kernel mediated control transfer; and

said step of enforcing includes ensuring that said kernel mediated control transfer complies with said rules.

39. A method according to claim 15, wherein:

said rules include different rules for handling a control transfer which is due to an exception and a control transfer which is due to an interrupt.

40. A method according to claim 15, wherein:

said step of enforcing includes restricting a plurality of said control transfers, including a control transfer which provides a direct call and a control transfer which provides an indirect call, said restricting implements a direct call-specific restriction for said control transfer which provides said direct call, and an indirect call-specific restriction for said control transfer which provides said indirect call.

41. A method according to claim 15, further comprising:

said step of enforcing includes restricting a plurality of said control transfers, including a control transfer which provides a jump and a control transfer which provides a return, said restricting implements a jump-specific restriction for said control transfer which provides said jump, and a return-specific restriction for said control transfer which provides said return.

53. A method according to claim 15, wherein:

the control transfers are of different types, including types of direct call, indirect call, return, conditional branch, direct jump, indirect jump and interrupt, and said set of rules is separately definable for each type of control transfer.

21. One or more processor readable storage devices having processor readable code embodied on said storage devices, said processor readable code for programming one or more processors to perform a method comprising:

analyzing control transfers of a program as said program executes on a computing system, said program comprising instructions, each instruction is native code of the one or more processors, said control transfers each being caused by execution of corresponding control transfer instructions causing a break in sequential flow of program execution, thereby determining a next instruction of the program to execute, said analyzing of the control transfers being performed prior to the execution of the control transfer instructions; and

ensuring that each said control transfer complies with a security policy, said security policy comprising a set of rules against which each of the control transfers is separately evaluated as part of the analyzing, the ensuring comprising announcing or handling a security violation when the control transfer instruction does not comply with the security policy.

22. One or more processor readable storage devices according to claim 21, wherein:

said enforcing of said security policy includes restricting execution privileges based on code origins, including determining whether code of said program is;

a) from an original image on disk and unmodified, b) dynamically generated but unmodified since generation or c) modified.

23. One or more processor readable storage devices according to claim 21, wherein said step of ensuring includes restricting at least one of said control transfer instructions based on instruction class.

24. One or more processor readable storage devices according to claim 21, wherein:

said enforcing of said security policy includes restricting control transfers based on instruction source and/or target.

25. One or more processor readable storage devices according to claim 21, wherein:

the control transfers are of different types, including types of direct call, indirect call, return, conditional branch, direct jump, indirect jump and interrupt, and said security policy includes different restrictions for the different types of control transfers.

26. One or more processor readable storage devices according to claim 21, wherein said step of ensuring includes requiring a pre-defined prologue before execution of a sequence of at least one of said instructions of the program and/or a pre-defined epilogue after execution of said sequence.

27. One or more processor readable storage devices according to claim 21, wherein:

said step of ensuring includes restricting execution privileges based on code origins, preventing bypass of sandboxing checks placed around a program operation and restricting the control transfers based on at least any one of source, destination or instruction class.