Deterring reverse-engineering of software systems by randomizing the siting of stack-based data

• US 7,210,134 B1
• Filed: 09/06/2002
• Issued: 04/24/2007
• Est. Priority Date: 09/06/2001
• Status: Active Grant

• Current Assignee: Rovi Technologies Corporation (Adeia Inc.)
• Original Assignee: Sonic Solutions (Adeia Inc.)
• Inventors: Langer, Randy
• Primary Examiner(s): An, Meng-Al T.
• Assistant Examiner(s): MITCHELL, JASON D
• Application Number:
  US10/237,515
• Time in Patent Office: 1,691 Days
• Field of Search: 717/127, 717/140, 713193-194, 712/202, 712/300
• US Class Current: 717/140
• CPC Class Codes: G06F 21/14 against software analysis or reverse engineering, e.g. by obfuscation
• CPC Class Codes: G06F 9/4484 Executing subprograms

Abstract

A given software process is composed on one or more threads of execution. Each thread possesses its own stack, a region of memory set aside by the operating system for that thread to store data. Popular programming languages rely heavily on stack-based data (frequently referred to as “local” or “automatic” data). It is a characteristic of deterministic machines like computers that, given the same problem to process with the same data, the same results, both intermediate and final, will result. This even extends to the sequence the software running on the computer will take to process the problem or data. This in turn means that for each thread making up the program, the data layout in the thread'"'"'s stack will be relatively consistent each time the program gets to a similar point in the processing of the problem and/or data. This represents a potential “point of repeatability” that a hacker can take advantage of. Embodiments of the current invention address this by introducing random amounts of “padding” into a thread'"'"'s stack, such that all data objects that exist “below” that point in the stack are offset by the amount of this random padding. A thread could have several points in its stack where the padding is introduced, resulting in better (more difficult to hack) randomization.

Claims

1. A method of managing memory for a thread of execution of a protected application in a computer system having at least one stack, the method comprising:

allocating a first amount of stack space for use by a first set of data adapted for use by the thread;

allocating a second amount of stack space, wherein the size of the second amount of stack space is randomly determined at runtime of the thread, wherein the second amount of stack space is not adapted for use by the thread, and wherein the allocation of the second amount of stack space occurs at runtime of the thread; and

allocating a third amount of stack space for use by a second set of data adapted for use by the thread, wherein the step of allocating the second amount of stack space occurs prior to one of the step of allocating the first amount of stack space and the step of allocating the third amount of stack space,

wherein the size of the second amount of stack space is randomly determined by using a recursive function that calls itself a random number of times.

2. The method of claim 1 wherein the recursive function introduces a frame each time that the recursive function calls itself thereby storing a plurality of frames in the stack, and wherein the plurality of frames is not adapted for use by the thread.

3. A computer system comprising:

a memory adapted to store a stack;

a processor coupled to the memory and adapted to execute a software routine of a protected application; and

programming logic adapted to be executed by the processor, said programming logic comprising;

means for allocating a first amount of stack space for use by a first set of data adapted for use by the software routine;

means for allocating a second amount of stack space, wherein the size of the second amount of stack space is randomly determined at runtime of the software routine, wherein the second amount of stack space is not adapted for use by the software routine, and wherein the allocation of the second amount of stack space occurs at runtime of the thread; and

means for allocating a third amount of stack space for use by a second set of data adapted for use by the software routine wherein the means for allocating the second amount of stack space allocates the second amount of stack space prior to a point in time when one of the first amount of stack space is allocated and the third amount of stack space is allocated,

wherein the size of the second amount of stack space is randomly determined by using a recursive function that calls itself a random number of times.

4. The system of claim 3 wherein the recursive function introduces a frame each time that the recursive function calls itself thereby storing a plurality of frames in the stack, and wherein the plurality of frames is not adapted for use by the software routine.