Direct call into system DLL detection system and method

• US 8,209,757 B1
• Filed: 06/27/2008
• Issued: 06/26/2012
• Est. Priority Date: 03/06/2008
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Kennedy, Mark, Pereira, Shane
• Primary Examiner(s): Barron, Jr., Gilberto
• Assistant Examiner(s): Cribbs, Malcolm
• Application Number:
  US12/163,747
• Time in Patent Office: 1,460 Days
• Field of Search: 726 22- 27
• US Class Current: 726/23
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine

Abstract

A method includes creating an intercept function for a tracked DLL function of a DLL being loaded into a suspicious module. Upon a determination that the tracked DLL function is invoked, a determination is made as to whether a return address of a caller of the tracked DLL function is within a legitimate return address range. The legitimate return address range includes an address range of the intercept function and excludes an address range of the suspicious module. If the return address is within the suspicious module, the suspicious module called the tracked DLL function directly. This indicates that the suspicious module is malicious and so protective action is taken.

Claims

1. A computer-implemented method comprising:

determining that one or more modules to be loaded into a process is suspicious, the determination of suspiciousness depending on one or more user-configurable factors, the determination resulting in one or more suspicious modules;

creating intercept functions for a tracked Dynamic Link Library (DLL) function of a Dynamic Link Library (DLL) being loaded into the one or more suspicious modules; and

determining whether said tracked DLL function is invoked, wherein upon a determination that said tracked DLL function is invoked, said method further comprising;

determining whether a return address of a caller of said tracked DLL function is within a legitimate return address range that corresponds to non-suspicious modules to determine whether said tracked DLL function is being directly called from said one or more suspicious modules, wherein in no legitimate event should said tracked DLL function be called directly from said one or more suspicious modules, said legitimate return address range comprising address ranges of said intercept functions, said legitimate return address range excluding address ranges of said one or more suspicious modules, and wherein said legitimate return address memory range comprises address ranges of said non-suspicious modules; and

dynamically updating said legitimate return address range, that corresponds to said non-suspicious modules and said intercept functions, when said non-suspicious modules and said intercept functions are created and terminated.

2. The computer-implemented method of claim 1 further comprising:

replacing an import address table entry for said tracked DLL function with an address of said intercept function.

3. The computer-implemented method of claim 1 further comprising hooking a prologue of said tracked DLL function.

4. The computer-implemented method of claim 3 wherein said hooking a prologue of said tracked DLL function comprises inserting a jump instruction into said tracked DLL function.

5. The computer-implemented method of claim 1 wherein said tracked DLL function is invoked when said caller calls said tracked DLL function.

6. The computer-implemented method of claim 1 wherein said return address is pushed on to a call stack when said tracked DLL function is called.

7. The computer-implemented method of claim 6 wherein said determining whether a return address of a caller of said tracked DLL function is within a legitimate return address range comprises reading said return address from said call stack.

8. The computer-implemented method of claim 7 wherein said return address is a pointer to a memory location where control is to be returned upon completion of said tracked DLL function.

9. The computer-implemented method of claim 1 further comprising determining said legitimate return address range comprising:

adding an address range of a non-suspicious module to said legitimate return address range upon a determination that said non-suspicious module is being created;

removing said address range of said non-suspicious module from said legitimate return address range upon a determination that said non-suspicious module is being terminated;

adding an address range of an intercept function to said legitimate return address range upon a determination that said intercept function is being created; and

removing said address range of said intercept function from said legitimate return address range upon a determination that said intercept function is being terminated.

10. The computer-implemented method of claim 1 wherein said legitimate return address range comprises any address except an address within said address ranges of said one or more suspicious modules.

11. The computer-implemented method of claim 10 further comprising determining said legitimate return address range comprising:

removing an address range of a suspicious module of the one or more suspicious modules from said legitimate return address range upon a determination that said suspicious module is being created; and

adding said address range of said suspicious module of the one or more suspicious modules to said legitimate return address range upon a determination that said suspicious module is being terminated.

12. The computer-implemented method of claim 1 wherein upon a determination that said return address is within said legitimate return address range, said method further comprising taking no further action.

13. The computer-implemented method of claim 1 wherein upon a determination that said return address is not within said legitimate return address range, said method further comprising taking protective action.

14. The computer-implemented method of claim 13 wherein said taking protective action comprises preventing said tracked DLL function from being executed.

15. The computer-implemented method of claim 13 wherein said taking protective action comprises defining said caller as a malicious module.

16. The computer-implemented method of claim 15 wherein said taking protective action further comprises restarting a general process comprising said malicious module and preventing said malicious module from being reloaded into said general process.

17. A computer system comprising:

a memory having stored therein a code module operating system (OS) interactions intercepting application; and

a processor coupled to said memory, wherein execution of said code module OS interactions intercepting application generates a method comprising;

determining that one or more modules to be loaded into a process is suspicious, the determination of suspiciousness depending on one or more user-configurable factors, the determination resulting in one or more suspicious modules;

creating intercept functions for a tracked Dynamic Link Library (DLL) function of a Dynamic Link Library (DLL) being loaded into the one or more suspicious modules; and

determining whether said tracked DLL function is invoked, wherein upon a determination that said tracked DLL function is invoked, said method further comprising;

determining whether a return address of a caller of said tracked DLL function is within a legitimate return address range that corresponds to non-suspicious modules to determine whether said tracked DLL function is being directly called from said one or more suspicious modules, wherein in no legitimate event should said tracked DLL function be called directly from said one or more suspicious modules, said legitimate return address range comprising address ranges of said intercept functions, said legitimate return address range excluding address ranges of said one or more suspicious modules, and wherein said legitimate return address memory range comprises address ranges of said non-suspicious modules; and

dynamically updating said legitimate return address range, that corresponds to said non-suspicious modules and said intercept functions, when said non-suspicious modules and said intercept functions are created and terminated.

18. A computer-program product comprising a tangible non-transitory computer readable storage medium containing computer program code comprising:

a code module operating system (OS) interactions intercepting application for, determining that one or more modules to be loaded into a process is suspicious, the determination of suspiciousness depending on one or more user-configurable factors, the determination resulting in one or more suspicious modules, the code module further for creating intercept functions for a tracked Dynamic Link Library (DLL) function of a Dynamic Link Library (DLL) being loaded into the one or more suspicious modules; and

said code module OS interactions intercepting application further for determining whether said tracked DLL function is invoked, wherein upon a determination that said tracked DLL function is invoked, said code module OS interactions intercepting application further for;

determining whether a return address of a caller of said tracked DLL function is within a legitimate return address range that corresponds to non-suspicious modules to determine whether said tracked DLL function is being directly called from said one or more suspicious modules, wherein in no legitimate event should said tracked DLL function be called directly from said one or more suspicious modules, said legitimate return address range comprising address ranges of said intercept functions, said legitimate return address range excluding address ranges of said one or more suspicious module, and wherein said legitimate return address memory range comprises address ranges of said non-suspicious modules; and

dynamically updating said legitimate return address range, that corresponds to said non-suspicious modules and said intercept functions, when said non-suspicious modules and said intercept functions are created and terminated.