METHOD AND APPARATUS FOR VERIFYING A SUSPECT RETURN POINTER IN A STACK

• US 20090144309A1
• Filed: 11/30/2007
• Published: 06/04/2009
• Est. Priority Date: 11/30/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cabrera Escandell, Marco A., Murray, Elizabeth J., McLane, Tommy L.
• Granted Patent: US 8,196,110 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs

Abstract

The present invention provides a computer implemented method, data processing system, and computer program product for verifying a return address. A computer stores the return address into a stack based on a function call. The computer generates a first hash based on a first stack frame and a second stack frame. The computer stores the first hash in a first canary location, wherein the first canary location is in the first stack frame. The computer executes at least one instruction of a routine referenced by the function call. The computer reads the first canary location to form a first suspect hash. The computer calculates a first verification hash based on the first stack frame and the second stack frame. The computer determines that the first verification hash matches the first suspect hash to form a first positive determination. The computer responsive to the first positive determination, the computer reads a second canary location to form a second suspect hash. The computer calculates a second verification hash based on the second stack frame. The computer determines that the second verification hash matches the second suspect hash to form a second positive determination. The computer responsive to the first positive determination and the second positive determination, the computer pops the return address off the stack. The computer executes at least one instruction at a memory location pointed to by the return address.

Claims

1. A computer implemented method for verifying a return address, the method comprising:

a) storing the return address into a stack based on a function call;

b) generating a first hash based on a first stack frame and a second stack frame;

c) storing the first hash in a first canary location, wherein the first canary location is in the first stack frame;

d) executing at least one instruction of a routine referenced by the function call;

e) reading the first canary location to form a first suspect hash;

f) calculating a first verification hash based on the first stack frame and the second stack frame;

g) determining that the first verification hash matches the first suspect hash to form a first positive determination;

h) responsive to the first positive determination, reading a second canary location to form a second suspect hash;

i) calculating a second verification hash based on the second stack frame;

j) determining that the second verification hash matches the second suspect hash to form a second positive determination;

k) responsive to the first positive determination and the second positive determination, popping the return address off the stack; and

l) executing at least one instruction at a memory location pointed to by the return address.

2. The computer implemented method of claim 1, wherein the second stack frame is an epilog flag different than the first stack frame, wherein the epilog flag is set at compile time.

3. The computer implemented method of claim 1, wherein storing the first hash comprises storing the return address into the first stack frame.

4. The computer implemented method of claim 1, wherein storing the first hash comprises storing the return address to a third stack frame, wherein the third stack frame is a top stack frame.

5. The computer implemented method of claim 1, wherein generating further comprises generating the first hash based on a hash of the first stack frame and a hash of the second stack frame.

6. The computer implemented method of claim 5, wherein the hash of the second stack frame is based on a hash of a stack frame of a parent routine of a routine associated with the second stack frame.

7. The computer implemented method of claim 6, wherein the first hash comprises storing the return address to a top stack frame, the parent routine is a prologue flag different than the top stack frame, and the prologue flag is at least one.

8. A data processing system comprising:

a bus;

a storage device connected to the bus, wherein computer usable code is located in the storage device;

a communication unit connected to the bus;

a processing unit connected to the bus, wherein the processing unit executes the computer usable code for verifying a return address, the processing unit further executes the computer usable code to store the return address into a stack based on a function call;

generate a first hash based on a first stack frame and a second stack frame;

store the first hash in a first canary location, wherein the first canary location is in the first stack frame;

execute at least one instruction of a routine referenced by the function call;

read the first canary location to form a first suspect hash;

calculate a first verification hash based on the first stack frame and the second stack frame;

determine that the first verification hash matches the first suspect hash to form a first positive determination;

responsive to the first positive determination, read a second canary location to form a second suspect hash;

calculate a second verification hash based on the second stack frame;

determine that the second verification hash matches the second suspect hash to form a second positive determination;

responsive to the first positive determination and the second positive determination, pop the return address off the stack; and

execute at least one instruction at a memory location pointed to by the return address.

9. The data processing system of claim 8, wherein the second stack frame is an epilog flag different than the first stack frame, wherein the epilog flag is set at compile time.

10. The data processing system of claim 8, wherein in executing the computer usable code to store the first hash, the data processing system stores the return address into the first stack frame.

11. The data processing system of claim 8, wherein in executing the computer usable code to store the first hash, the data processing system stores the return address to a third stack frame, wherein the third stack frame is a top stack frame.

12. The data processing system of claim 8, wherein in executing the computer usable code to generate, the data processing system generates the first hash based on a hash of the first stack frame and a hash of the second stack frame.

13. The data processing system of claim 12, wherein the hash of the second stack frame is based on a hash of a stack frame of a parent routine of a routine associated with the second stack frame.

14. The data processing system of claim 13, wherein the first hash comprises storing the return address to a top stack frame, the parent routine is a prologue flag different than the top stack frame, and the prologue flag is at least one.

15. A computer program product the computer program product comprising:

a computer usable medium having computer usable program code for verifying a return address, said computer program product including;

computer usable program code for storing the return address into a stack based on a function call;

computer usable program code for generating a first hash based on a first stack frame and a second stack frame;

computer usable program code for storing the first hash in a first canary location, wherein the first canary location is in the first stack frame;

computer usable program code for executing at least one instruction of a routine referenced by the function call;

computer usable program code for reading the first canary location to form a first suspect hash;

computer usable program code for calculating a first verification hash based on the first stack frame and the second stack frame;

computer usable program code for determining that the first verification hash matches the first suspect hash to form a first positive determination;

computer usable program code for responsive to the first positive determination, reading a second canary location to form a second suspect hash;

computer usable program code for calculating a second verification hash based on the second stack frame;

computer usable program code for determining that the second verification hash matches the second suspect hash to form a second positive determination;

computer usable program code for responsive to the first positive determination and the second positive determination, popping the return address off the stack; and

computer usable program code for executing at least one instruction at a memory location pointed to by the return address.

16. The computer program product of claim 15, wherein the second stack frame is an epilog flag different than the first stack frame, wherein the epilog flag is set at compile time.

17. The computer program product of claim 15, wherein the computer usable program code for storing the first hash comprises computer usable program code for storing the return address into the first stack frame.

18. The computer program product of claim 15, wherein the computer usable program code for storing the first hash comprises computer usable program code for storing the return address to a third stack frame, wherein the third stack frame is a top stack frame.

19. The computer program product of claim 15, wherein computer usable program code for generating further comprises computer usable program code for generating the first hash based on a hash of the first stack frame and a hash of the second stack frame.

20. The computer program product of claim 19, wherein the hash of the second stack frame is based on a hash of a stack frame of a parent routine of a routine associated with the second stack frame.