System and method for preventing race condition vulnerability

• US 8,127,413 B2
• Filed: 07/11/2006
• Issued: 03/06/2012
• Est. Priority Date: 07/11/2006
• Status: Expired due to Fees

• Current Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Original Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Inventors: Pu, Calton, Wei, Jinpeng
• Primary Examiner(s): Orgad, Edan
• Assistant Examiner(s): TURCHEN, JAMES R
• Application Number:
  US11/484,976
• Publication Number:
  US 20080016410A1
• Time in Patent Office: 2,065 Days
• Field of Search: 726 22- 26
• US Class Current: 26/25
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2221/2105 Dual mode as a secondary aspect

Abstract

A method for identifying vulnerable system call pairs is disclosed. The method is based on a model for identifying Time-Of-Check-To-Time-Of-Use (TOCTTOU) problem (called STEM), which enumerates the potential file system call pairs (called exploitable TOCTTOU pairs) that form the check/use steps. The system function calls are classified into a plurality of predefined classes and pairs of the function calls are formed according to predefined criteria, where the function calls within a pair are associated with the same file invariant.

Claims

1. A method for reducing vulnerability in a computer system, the computer system having a generation unit, a timer and memory, by identifying vulnerable pairs of function calls, comprising the steps of:

generating, by the generation unit, a plurality of pairs of the function calls according to a predefined criteria;

invoking a first function call from a pair of the function calls by a first user;

creating a file invariant associated with the pair of the function calls, the file invariant having a tainted flag;

starting the timer;

associating the timer with the file invariant;

if a second function call is from the pair of the function calls and is invoked by a second user and the timer has not expired, denying the second function call to the second user;

if the second function call is from the pair of the function calls and is invoked by the second user and the timer has expired, setting the tainted flag for the file invariant and allowing the second function call by the second user;

if the second function call is from the pair of the function calls and is invoked by the first user and the tainted flag for the file invariant is set, denying the second function call to the first user; and

if the second function call is from the pair of the function calls and is invoked by the first user and the tainted flag for the file invariant is not set, allowing the second function call to the first user, removing the file invariant, and stopping the timer.

2. The method of claim 1, further comprising the step of indicating an alert if the tainted flag for the file invariant is set.

3. The method of claim 1 further comprising the step of setting a flag in the file invariant if the information from the second functional call is different from the file invariant.

4. The method of claim 1, wherein the step of generating a plurality of pairs further comprising the step of pairing a function call from a first predefined class with a function call from a second predefined class.

5. The method of claim 1, wherein the file invariant being a link between a file name and a file storage location.

6. The method of claim 1, wherein the file invariant being a file ownership information.

7. The method of claim 1, further comprising the step of classifying the function calls into to a plurality of predefined classes, wherein the plurality of predefined classes include creation, removal, check, and normal use.

8. The method of claim 1, wherein the plurality of pairs being a full list of potential combination of pairs of function calls that conform to the predefined criteria.

9. The method of claim 1, wherein the step of disabling the file invariant further comprising the step of setting the tainted flag.

10. The method of claim 1, further comprising the step of disabling the file invariant further comprising the step of setting the tainted flag.

11. The method of claim 1, further comprising the step of, if the second function call is part of a sequence of at least three function calls and is invoked by the first user and the tainted flag for the file invariant is not set, resetting the timer and allowing the second function call to the first user.

12. A system for reducing vulnerability in a computer system by identifying vulnerable pairs of function calls, comprising:

a classifying unit for classifying the function calls into a plurality of predefined classes; and

a generation unit for generating a plurality of pairs of the function calls according to a predefined criteria;

wherein

a file invariant is created by the computer system and associated with a pair of the function calls generated by the generation unit when a first function call from the pair of the function calls is invoked by a first user, the file invariant includes a tainted flag, the tainted flag is set when the pair of function calls being invoked by two different processes, the file invariant is associated with a timer and the timer is set when the first function call from the pair of the function calls is invoked by a first user,

if the timer is expired and a second function call in the pair of the function calls is invoked by a second user, allowing the second function call by the second user and setting the tainted flag, and

if the second function call in the pair of the function calls is invoked by the first user and the tainted flag is not set, allowing the second function call by the first user, removing the file invariant, and stopping the timer.

13. The system of claim 12, further comprising a file invariant creation unit for creating the file invariant.

14. The system of claim 13, further comprising a file invariant checking unit for checking information created when the second function call is invoked against the first file invariant.

15. The system of claim 12, wherein the generation unit further being capable of pairing a function call from a first predefined class with a function call from a second predefined class.

16. The system of claim 12, wherein the file invariant being a link between a file name and a file storage location.

17. The system of claim 12, wherein the file invariant being a file ownership information.

18. The system of claim 12, wherein the plurality of predefined classes include creation, removal, check, and normal use.

19. The system of claim 12, wherein, if the second function call is part of a sequence of at least three function calls and is invoked by the first user and the tainted flag for the file invariant is not set, resetting the timer and allowing the second function call to the first user.

20. A computer readable storage device having computer readable code, which when executed by a computer implements an operating system for a computer system with reduced vulnerability, the operating system comprising:

a library of plurality of function calls;

a classifying unit, the classifying unit classifies the function calls into a plurality of predefined classes; and

a generation unit, the generation unit generates a plurality of pairs of the function calls according to a predefined criteria,

wherein

each pair of the function calls being associated with a file invariant when a first function call from the pair of the function calls is invoked by a first user, the file invariant includes a tainted flag, the tainted flag is set when a pair of function calls being invoked by different processes, the file invariant is associated with a timer and the timer is set when the first function call from the pair of the function calls is invoked by the first user,

if the timer is expired and a second function call in the pair of the function calls is invoked by a second user, allowing the second function call by the second user and setting the tainted flag, and

if the second function call in the pair of the function calls is invoked by the first user and the tainted flag is not set, allowing the second function call by the first user, removing the file invariant, and stopping the timer.

21. The operating system of claim 20, further comprising a file invariant creation unit for creating a file invariant when the first function call within a pair of the function calls is invoked.

22. The operating system of claim 21, further comprising a file invariant checking unit for checking information created when the second function call is invoked against the first file invariant.

23. The operating system of claim 20, wherein the generation unit further being capable of pairing a function call from a first predefined class with a function call from a second predefined class.

24. The operating system of claim 20, wherein the file invariant being a link between a file name and a file storage location.

25. The operating system of claim 20, wherein the file invariant being a file ownership information.

26. The operating system of claim 20, wherein the plurality of predefined classes include creation, removal, check, and normal use.

27. The operating system of claim 20, wherein, if the second function call is part of a sequence of at least three function calls and is invoked by the first user and the tainted flag for the file invariant is not set, resetting the timer and allowing the second function call to the first user.