Shell code blocking system and method

• US 20040158729A1
• Filed: 02/06/2003
• Published: 08/12/2004
• Est. Priority Date: 02/06/2003
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Szor, Peter
• Granted Patent: US 7,228,563 B2
• Time in Patent Office: Days
• US Class Current: 713/200
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow

Abstract

A method includes hooking a critical operating system function, originating a call to the critical operating system function with a call module of a parent application, stalling the call, determining a location of the call module in memory, and determining whether the location is in an executable area of the memory. Upon a determination that the call module is not in the executable area, the method further includes terminating the call. By terminating the call, execution of a child application that would otherwise allow unauthorized remote access is prevented.

Claims

1. A method comprising:

originating an operating system function call with a call module; and

determining whether said call module is in an executable area of memory.

2. The method of claim 1 further comprising determining that said call module is not in said executable area during said determining.

3. The method of claim 2 further comprising terminating said operating system function call.

4. The method of claim 3 further comprising notifying a user or administrator that said operating system function call has been terminated.

5. The method of claim 2 further comprising terminating a parent application comprising said call module.

6. The method of claim 5 further comprising notifying a user or administrator that said parent application has been terminated.

7. The method of claim 1 further comprising determining that said call module is in said executable area during said determining.

8. The method of claim 1 further comprising stalling said operating system function call.

9. The method of claim 8 further comprising:

determining that said call module is in said executable area during said determining; and

allowing said operating system function call to proceed.

10. The method of claim 1 wherein said executable area consists of a page that is marked as executable.

11. A method comprising:

hooking at least a first operating system function;

stalling a call originating from a call module to said first operating system function;

determining a location of said call module in memory; and

determining whether said location is in an executable area of said memory, wherein upon a determination that said call module is not in said executable area, said method further comprising terminating said call.

12. The method of claim 11 wherein upon a determination that said call module is in said executable area, said method further comprising allowing said call to proceed.

13. The method of claim 12 wherein said call module is non-malicious.

14. The method of claim 11 wherein upon a determination that said call module is not in said executable area, said method further comprising terminating a parent application comprising said call module.

15. The method of claim 14 wherein said parent application comprises malicious code.

16. The method of claim 11 wherein said determining a location of said call module in memory comprises analyzing a content of a stack.

17. The method of claim 16 wherein said content is analyzed to locate a return pointer pointing to said call module.

18. The method of claim 11 wherein said hooking comprises redirecting calls to said first operating system function to a hook module.

19. The method of claim 18 wherein said redirecting calls is performed with a hooked system service table.

20. A computer system comprising:

a means for hooking at least a first operating system function;

a means for stalling a call originating from a call module to said first operating system function;

a means for determining a location of said call module in memory; and

a means for determining whether said location is in an executable area of said memory, wherein upon a determination that said call module is not in said executable area, said method further comprising terminating said call.

21. A computer-program product comprising a computer-readable medium containing computer program code comprising:

a shell code blocking application for determining whether a call module originating an operating system function call is in an executable area of memory.

22. The computer-program product of claim 21 wherein said shell code blocking application is further for determining that said call module is not in said executable area during said determining.

23. The computer-program product of claim 22 wherein said shell code blocking application is further for terminating said operating system function call.

24. The computer-program product of claim 23 wherein said shell code blocking application is further for notifying a user or administrator that said operating system function call has been terminated.

25. The computer-program product of claim 22 wherein said shell code blocking application is further for terminating a parent application comprising said call module.

26. The computer-program product of claim 25 wherein said shell code blocking application is further for notifying a user or administrator that said parent application has been terminated.

27. The computer-program product of claim 21 wherein said shell code blocking application is further for determining that said call module is in said executable area during said determining.

28. The computer-program product of claim 21 wherein said shell code blocking application is further for stalling said operating system function call.

29. The computer-program product of claim 28 wherein said shell code blocking application is further for:

determining that said call module is in said executable area during said determining; and

allowing said operating system function call to proceed.

30. The computer-program product of claim 21 wherein said executable area consists of a page that is marked as executable.