Pointguard: method and system for protecting programs against pointer corruption attacks

• US 7,752,459 B2
• Filed: 12/06/2002
• Issued: 07/06/2010
• Est. Priority Date: 12/06/2001
• Status: Expired due to Fees

• Current Assignee: RPX Corporation
• Original Assignee: Novell Incorporated (Open Text Corporation)
• Inventors: Beattie, Steven Michael, Cowan, Stanley Crispin, Wagle, Perry Michael, Arnold, Seth Richard
• Primary Examiner(s): Homayounmehr, Farid
• Application Number:
  US10/313,940
• Publication Number:
  US 20030182572A1
• Time in Patent Office: 2,769 Days
• Field of Search: 713/190
• US Class Current: 713/190
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 21/125 by manipulating the program code, e.g. source code, compiled code, interpreted code, machine code
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 9/4486 Formation of subprogram jump address

Abstract

To protect computer programs against security attacks that attempt to corrupt pointers within the address space of the program, the value of a pointer is encrypted each time the pointer is initialized or modified, and then the value is decrypted before use, i.e., each time the pointer is read. Preferably, the encrypting and decrypting steps are effected by instructions generated by a compiler during compilation of the program. One convenient method of implementing the encrypting and decrypting steps is by XOR'"'"'ing the pointer with a predetermined encryption key value, which could be specially selected or selected at random.

Claims

1. A method of protecting an application computer program during execution outside of a special environment against pointer corruption attacks, the application computer program implemented using the C programming language, the method comprising the steps of:

identifying a function;

implementing a protected version of the function, the protected version of the function including;

encrypting values of all data pointers each time the values of the data pointers are modified; and

decrypting the encrypted value of at least one data pointer before use each time the data pointer is read; and

exporting the protected version of the function and an unprotected version of the function,

wherein said encrypting and decrypting steps are effected by instructions generated by a compiler during compilation of the said program.

2. A method according to claim 1 wherein said encrypting and decrypting steps are effected by XOR'"'"'ing the data pointer with a predetermined encryption key value.

3. A method according to claim 2 including selecting the key value at random.

4. A method according to claim 2, further comprising storing the encryption key in read-only memory to protect the encryption key from tampering.

5. A method according to claim 1 including storing the data pointer values in the clear only in CPU registers and storing the encrypted values of the data pointers in addressable memory.

6. A method according to claim 1 wherein instructions to implement said encryption and decryption steps are inserted into the program in a preprocessor.

7. A method according to claim 1 wherein instructions to implement said encryption and decryption steps are inserted by a compiler during manipulation of an intermediate representation of the program.

8. A method according to claim 1 wherein instructions to implement said encryption and decryption steps are inserted by a compiler in the context of an architecture-dependent representation by transforming instructions to load data pointer values from memory into registers to add the decryption instructions, and transform the saving of data pointer values from registers to memory to add the encryption instructions.

9. A method of protecting an application computer program during execution outside of a special environment against pointer corruption attacks, the application computer program implemented using the C programming language, the method comprising the steps of:

identifying a function;

implementing a protected version of the function, the protected version of the function including;

encrypting values of pointers each time the values of the pointers are modified; and

decrypting the encrypted value of at least one pointer before use each time the pointer is read; and

exporting the protected version of the function and an unprotected version of the function,

wherein said encrypting and decrypting steps are effected by instructions generated by a compiler during compilation of the said program.

10. A method according to claim 9 wherein said encrypting and decrypting steps are effected by XOR'"'"'ing the pointer with a predetermined encryption key value.

11. A method according to claim 10 including selecting the key value at random.

12. A method according to claim 10 further comprising storing the encryption key in read-only memory to protect the encryption key from tampering.

13. A method according to claim 9 including storing the pointer values in the clear only in CPU registers and storing the encrypted values of the pointers in addressable memory.

14. A method according to claim 9 wherein instructions to implement said encryption and decryption steps are inserted into the program in a preprocessor.

15. A method according to claim 9 wherein instructions to implement said encryption and decryption steps are inserted by a compiler during manipulation of an intermediate representation of the program.

16. A method according to claim 9 wherein instructions to implement said encryption and decryption steps are inserted by a compiler in the context of an architecture-dependent representation by transforming instructions to load pointer values from memory into registers to add the decryption instructions, and transform the saving of pointer values from registers to memory to add the encryption instructions.