Method to identify buffer overflows and RLIBC attacks

• US 7,945,953 B1
• Filed: 07/06/2005
• Issued: 05/17/2011
• Est. Priority Date: 07/06/2005
• Status: Active Grant

• Current Assignee: Gen Digital Inc.
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Salinas, Govind, Conover, Matthew, Satish, Sourabh
• Primary Examiner(s): Srivastava; Vivek
• Assistant Examiner(s): BROWN, ANTHONY D
• Application Number:
  US11/176,855
• Time in Patent Office: 2,141 Days
• Field of Search: 726 22- 26, 713187-191
• US Class Current: 726/22
• CPC Class Codes: G06F 12/1441 for a range
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security

Abstract

A method and system detect buffer overflows and RLIBC attacks by determining if a critical call initiating function is a “potential threat”. In one embodiment, a critical call initiating function is considered a potential threat if the value of the return address of the critical call initiating function points to a location in memory between the location of the highest Thread Environment Block (TEB) or Process Environment Block (PEB) and the location of the lowest Thread Environment Block (TEB) or PEB. In another embodiment, a critical call initiating function making a call to a predefined critical operating system function is considered a potential threat if the value of the return address of the critical call initiating function points to the beginning of a new function with a zero offset.

Claims

1. A computer system implemented method for blocking a buffer overflow comprising:

a computer system;

a memory associated with the computer system;

a processor associated with the computer system, the processor associated with the computer system executing instructions for implementing at least part of the computer system implemented method for blocking a buffer overflow, the computer system implemented method for blocking a buffer overflow comprising;

stalling a call to a critical operating system (OS) function, said call to a critical operating system (OS) function being made by a critical call initiating function residing in the memory associated with the computer system;

determining whether a value of a return address of said critical call initiating function points to a location in said memory associated with the computer system that corresponds to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system;

taking protective action to protect the computer system upon a determination that said return address of said critical call initiating function does point to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system; and

allowing said call to a critical operating system (OS) function to proceed upon a determination that said return address of said critical call initiating function does not point to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system.

2. The computer system implemented method for blocking a buffer overflow of claim 1 wherein said taking protective action comprises terminating said call to a critical operating system (OS) function.

3. A system comprising:

a computer system;

a memory associated with the computer system;

a processor associated with the computer system,

means for stalling a call to a critical operating system (OS) function, said call to a critical operating system (OS) function being made by a critical call initiating function residing in the memory associated with the computer system;

means for determining whether a value of a return address of said critical call initiating function points to a location in said memory associated with the computer system that corresponds to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system;

means for taking protective action to protect the computer system upon a determination that said return address of said critical call initiating function does point to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system; and

means for allowing said call to a critical operating system (OS) function to proceed upon a determination that said return address of said critical call initiating function does not point to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system.

4. The system of claim 3 wherein said taking protective action comprises terminating said call to a critical operating system (OS) function.

5. A computer system implemented method for blocking a buffer overflow comprising:

a computer system;

a memory associated with the computer system;

a processor associated with the computer system, the processor associated with the computer system executing instructions for implementing at least part of the computer system implemented method for blocking a buffer overflow, the computer system implemented method for blocking a buffer overflow comprising;

stalling a call to a critical operating system (OS) function, the critical call operating system function being an operating system function necessary for a first application to cause execution of a second application, said call to a critical operating system (OS) function being made by a critical call initiating function residing in a the memory associated with the computer system;

determining whether a value of a return address of said critical call initiating function points to a location in said memory associated with the computer system that corresponds to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system;

taking protective action to protect the computer system upon a determination that said return address of said critical call initiating function does point to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system; and

allowing said call to a critical operating system (OS) function to proceed upon a determination that said return address of said critical call initiating function does not point to a location in a Thread Environment Block (TEB) or a Process Environment Block (PEB) of said memory associated with the computer system.