Method and apparatus for protecting memory stacks

• US 6,996,677 B2
• Filed: 02/20/2003
• Issued: 02/07/2006
• Est. Priority Date: 11/25/2002
• Status: Active Grant

• Current Assignee: Ciena Corporation
• Original Assignee: Nortel Networks Limited (Nortel Networks Corporation)
• Inventors: Lee, Michael C., Dobranski, Lawrence
• Primary Examiner(s): Sparks, Donald
• Assistant Examiner(s): Peugh, Brian R.
• Application Number:
  US10/368,596
• Publication Number:
  US 20040103252A1
• Time in Patent Office: 1,083 Days
• Field of Search: 713/187, 713/189, 713/190, 713/193, 712/242, 712/34, 712/35, 711/132, 711/154, 711/4, 710/56
• US Class Current: 711/132
• CPC Class Codes: G06F 12/1441 for a range
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 21/71 to assure secure computing or processing of information

Abstract

Method and apparatus for protecting processing elements from buffer overflow attacks are provided. The apparatus includes a memory stack for, upon execution of a jump to subroutine, storing a return address in a first location in a stack memory. A second location separate from the stack memory for storing an address of the first location and a third location separate from the stack memory for storing the return address itself are included. A first comparator upon completion of the subroutine, compares the address stored in the second location to the first location in the stack memory and a first interrupt generator provides an interrupt signal if locations are not the same. A second comparator looks at the return address stored in the third location and the return address stored in the first location in the stack memory and has a second interrupt generator for generating an interrupt signal if addresses are not the same. A further method and apparatus for protecting processing elements from buffer overflow attacks includes a memory stack for, upon execution of a jump to subroutine in a first processor, storing a return address in a first location in a stack memory and a second location separate from the stack memory for storing results for the subroutine operation. Also included is a second processor including routines for data manipulation associated with the subroutine, separate from the first processor and for storing any resultant data in the second location, which is readable by the first processor separate from the stack memory.

Claims

1. A method of protecting processing elements from buffer overflow attacks, the method comprising the steps of:

upon execution of a jump to subroutine, storing a return address in a first location in a stack memory;

storing an address of the first location in a second location separate from the stack memory;

storing the return address itself in a third location separate from the stack memory;

upon completion of the subroutine, comparing the address stored in the second location to the first location in the stack memory;

if equal, comparing the return address stored in the third location to the return address stored in the first location in the stack memory; and

if equal, returning to the return address.

2. A method as claimed in claim 1 wherein if the address stored in the second location and the first location address in the stack memory are not equal, generating a first interrupt signal to the processing element.

3. A method as claimed in claim 1 wherein if the return address stored in the third location and contents of the first location in the stack memory are not equal, generating a second interrupt signal to the processing element.

4. Apparatus for protecting processing elements from buffer overflow attacks, comprising:

a memory stack for, upon execution of a jump to subroutine, storing a return address in a first location in a stack memory;

a second location separate from the stack memory for storing an address of the first location;

a third location separate from the stack memory for storing the return address itself;

a first comparator for, upon completion of the subroutine, comparing the address stored in the second location to the first location in the stack memory and having a first interrupt generator for generating an interrupt signal if locations are not the same; and

a second comparator for comparing the return address stored in the third location to the return address stored in the first location in the stack memory and having a second interrupt generator for generating an interrupt signal if addresses are not the same.

5. A method of protecting processing elements from buffer overflow attacks, the method comprising the steps of:

upon execution of a jump to subroutine, storing a return address in a first location in a stack memory;

storing the return address itself in a second location separate from the stack memory;

comparing the return address stored in the second location to the return address stored in the first location in the stack memory; and

if equal, returning to the return address.

6. A method as claimed in claim 5 wherein if the address stored in the second location and the first location in the stack memory are not equal, generating an interrupt signal to the processing element.

7. Apparatus for protecting processing elements from buffer overflow attacks, comprising:

a memory stack for, upon execution of a jump to subroutine, storing a return address in a first location in a stack memory;

a second location separate from the stack memory for storing the return address itself;

a comparator for comparing the return address stored in the second location to the return address stored in the first location in the stack memory; and

an interrupt generator for generating an interrupt signal if addresses are not the same.

8. A method of protecting processing elements from buffer overflow attacks, the method comprising the steps of:

upon execution of a jump to subroutine in a first processor, storing a return address in a first location in a stack memory;

processing a data manipulation associated with the subroutine in a second processor, separate from the main processor and storing any resultant data in a second location, which is readable by the first processor separate from the stack memory;

upon completion of the subroutine, returning control to the first processor for reading of the return address stored in the first location in the stack memory.

9. A method as claimed in claim 8 wherein the data manipulation includes one of a string processing, buffer manipulation procedure, and buffer manipulation function.

10. A method as claimed in claim 9 wherein the string processing is a string copy.

11. A method as claimed in claim 9 wherein parameters associated with the procedure or function are placed on the stack memory of the first processor stack.

12. A method as claimed in claim 8 wherein the step of processing is initiated by executing an operation code to transfer control to interface routines of the second processor.

13. A method as claimed in claim 8 wherein the step of processing includes allocating internal memory in the second processor for storing results.

14. A method as claimed in claim 13 wherein the step of converting a called string or buffer parameters into an internal string/buffer representation with finite, defined length.

15. Apparatus for protecting processing elements from buffer overflow attacks, comprising:

a memory stack for, upon execution of a jump to subroutine in a first processor, storing a return address in a first location in a stack memory;

a second location separate from the stack memory for storing results for the subroutine operation;

a second processor including routines for data manipulation associated with the subroutine, separate from the first processor and for storing any resultant data in the second location, which is readable by the first processor separate from the stack memory.