Security system with methodology for interprocess communication control

• US 8,136,155 B2
• Filed: 09/12/2003
• Issued: 03/13/2012
• Est. Priority Date: 04/01/2003
• Status: Active Grant

• Current Assignee: Check Point Software Technologies Incorporated (Check Point Software Technologies Limited)
• Original Assignee: Check Point Software Technologies Incorporated (Check Point Software Technologies Limited)
• Inventors: Freund, Gregor P.
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): TRUVAN, LEYNNA THANH
• Application Number:
  US10/605,189
• Publication Number:
  US 20040199763A1
• Time in Patent Office: 3,105 Days
• Field of Search: 726/1, 726/5, 726/14, 726 18- 19, 726 22- 27, 713/152, 713/167, 709/229, 709310-313, 709223-225, 707/687, 707/694, 719/126, 719/127, 717/131, 717171-174, 717176-178
• US Class Current: 726/22
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: H04L 63/104 Grouping of entities

Abstract

A security system with methodology for interprocess communication control is described. In one embodiment, a method for controlling interprocess communication is provided that includes steps of: defining rules indicating which system services a given application can invoke; trapping an attempt by a particular application to invoke a particular system service; identifying the particular application that is attempting to invoke the particular system service; and based on identity of the particular application and on the rules indicating which system services a given application can invoke, blocking the attempt when the rules indicate that the particular application cannot invoke the particular system service.

Claims

1. In a computer system operating under control of an operating system, a method for detecting and preventing indirect access to a computer network by potentially malicious applications already installed and executing on the computer system, the method comprising:

defining rules governing access by applications on the computer system to the computer network including rules indicating which system services of the operating system are monitored for detecting and preventing indirect access to the computer network by potentially malicious applications that are already installed and executing on the computer system, but which are capable of obtaining indirect access to the computer network through system services;

trapping an attempt by a particular application already installed and executing on the computer system to gain indirect access to the computer network through invocation of a particular system service being monitored, wherein said trapping includes rerouting the attempt to invoke the particular system service from a system dispatch table to an interprocess communication controller by replacing an original destination address in the system dispatch table with an address of the interprocess communication controller;

detecting based on the rules governing access by applications to the computer network if the attempt to invoke the particular system service by the particular application rerouted to the interprocess communication controller constitutes an unauthorized attempt by a potentially malicious application already installed and executing on the computer system to obtain indirect access to the computer network by invoking the particular system service which in turn accesses the computer network on behalf of the potentially malicious application; and

if the attempt to invoke the particular system service constitutes an unauthorized attempt by a potentially malicious application to access the computer network indirectly, preventing the potentially malicious application from obtaining indirect access to the computer network by blocking the attempt.

2. The method of claim 1, wherein said trapping step includes intercepting operating system calls for invoking the particular system service.

3. The method of claim 1, wherein said trapping step includes intercepting local procedure calls for invoking the particular system service.

4. The method of claim 1, wherein said trapping step includes intercepting an attempt to open a communication channel to the particular system service.

5. The method of claim 1, further comprising the steps of:

retaining the original destination address; and

using the original destination address for invoking the particular system service if the interprocess communication controller determines not to block the attempt.

6. The method of claim 1, wherein the rules governing access by applications on the computer system to the computer network including rules specifying which system services are monitored are established based on user input.

7. The method of claim 1, wherein the step of blocking the attempt is based upon consulting a rules engine for determining applications authorized to invoke the particular system service to indirectly access the computer network.

8. The method of claim 1, wherein the step of preventing includes obtaining user input as to whether the unauthorized application should now be authorized to invoke the particular system service.

9. The method of claim 8, wherein said step of obtaining user input includes the substeps of:

providing information to the user about which particular application is attempting to invoke the particular system service; and

receiving user input as to whether the particular application should be blocked from invoking the particular system service.

10. A non-transitory computer-readable medium having computer-executable instructions for performing the method of claim 1.

11. The method of claim 1, further comprising:

downloading a set of computer-executable instructions for performing the method of claim 1.

12. In a computer system operating under control of an operating system, a method for detecting and preventing indirect access to the Internet by potentially malicious applications already installed and executing on the computer system, the method comprising:

defining a policy indicating which system services of the operating system are monitored for detecting and preventing indirect access to the Internet by potentially malicious applications that are already installed and executing on the computer system but which are capable of obtaining indirect access to the Internet through system services, said policy specifying processes authorized to access the Internet;

intercepting an attempt by a first process to communicate with a second process in a manner that provides the first process with indirect access to Internet, wherein said intercepting includes rerouting the attempt by the first process to communicate with the second process from a system dispatch table to an interprocess communication controller by replacing an original destination address in the system dispatch table with an address of the interprocess communication controller;

identifying the first process that is attempting to communicate with the second process;

identifying the second process;

based on said policy, determining whether the first process may communicate with the second process in a manner that provides the first process with indirect access to Internet, including determining if the attempt to invoke the particular system service rerouted to the interprocess communication controller does constitute an unauthorized attempt by a potentially malicious application to access the Internet indirectly; and

allowing the first process to communicate with the second process if said policy indicates that the first process may communicate with the second process in a manner that provides the first process with indirect access to Internet and does not constitute an unauthorized attempt by a potentially malicious application to access the Internet indirectly.

13. The method of claim 12, wherein the first process comprises an instance of an application program.

14. The method of claim 12, wherein the second process comprises a system service.

15. The method of claim 12, wherein said intercepting step includes intercepting operating system calls made by the first process to attempt to communicate with the second process.

16. The method of claim 12, wherein said intercepting step includes detecting local procedure calls.

17. The method of claim 12, wherein said intercepting step includes detecting an attempt by the first process to open a communication channel to the second process.

18. The method of claim 12, wherein said step of identifying the second process includes evaluating parameters of the attempt made by the first process to communicate with the second process.

19. The method of claim 12, wherein said policy specifies particular processes to be protected from communications made by other processes.

20. The method of claim 12, further comprising:

providing for a process to be registered in order to be protected from communications made by other processes; and

determining whether to allow the first process to communicate with the second process based, at least in part, upon determining whether the second process is registered.

21. The method of claim 20, wherein said determining step is based, at least in part, on the type of communication the first process is attempting with the second process.

22. In a computer system operating under control of an operating system, a method for detecting and preventing one application already installed and executing on the computer system from gaining indirect Internet access through other applications, the method comprising:

registering a first application to be protected from serving as a proxy by which other applications may gain indirect Internet access for detecting and preventing indirect access to the Internet by potentially malicious applications that are already installed and executing on the computer system but which are capable of obtaining indirect access to the Internet through said first application;

detecting an attempt by a second application already installed and executing on the computer system to access the first application for purposes of using the first application as a proxy for indirect Internet access;

identifying the second application that is attempting to access the first application for purposes of using the first application as a proxy for indirect Internet access; and

rerouting the attempt to access the first application through an interprocess communication controller that determines whether to allow the attempt, based on rules indicating whether the second application is authorized to access the first application using interprocess communication, wherein said rerouting includes replacing an original destination address for the first application in a system dispatch table with an address of the interprocess communication controller.

23. The method of claim 22, wherein said registering step includes supplying rules specifying particular communications from which the first application is to be protected.

24. The method of claim 23, wherein the interprocess communication controller determines whether to allow the attempt based, at least in part, upon the rules specifying particular communications from which the first application is to be protected.

25. The method of claim 22, wherein said detecting step includes intercepting operating system calls for accessing the first application.

26. The method of claim 22, wherein said detecting step includes detecting a graphical device interface (GDI) message sent to the first application.

27. The method of claim 26, wherein said identifying step includes evaluating parameters of the message sent to the first application.

28. The method of claim 22, wherein said detecting step includes detecting an attempt to send keystoke data to a window of the first application.

29. The method of claim 22, wherein said detecting step includes detecting an attempt to send mouse movement data to a window of the first application.

30. The method of claim 22, wherein said rules indicating whether the second application may access the first application includes rules indicating particular types of communications which are allowed.

31. The method of claim 22, further comprising:

if the interprocess communication controller allows the attempt to access the first application, routing the attempt to the first application.

32. A system for detecting and preventing indirect access to the Internet access by controlling interprocess communication between applications, the system comprising:

a computer having at least one processor, said computer operating under control of an operating system providing interprocess communication;

a policy specifying applications that are permitted to communicate with a first application using interprocess communication, said first application capable of providing indirect Internet access to other applications, so as to detect and prevent indirect access to the Internet by potentially malicious applications that are already installed and executing on the computer system but which are capable of obtaining indirect access to the Internet through said first application;

a module for detecting an attempt by a second application already installed and executing on the computer system to gain indirect Internet access through the first application using interprocess communication and reroutes the attempt from a dispatch table to an interprocess communication controller by replacing an address of the first application in the dispatch table with an address of the interprocess communication controller; and

an interprocess communication controller for identifying the second application attempting to gain indirect Internet access through the first application using interprocess communication and determining whether to permit the communication based upon the identification of the second application and the policy specifying applications permitted to communicate with the first application.

33. The system of claim 32, wherein said policy includes rules indicating particular types of communications which are permitted.

34. The system of claim 32, further comprising:

a rules engine for specifying applications that are permitted to communicate with the first application using interprocess communication.

35. The system of claim 32, further comprising:

a registration module for establishing said policy.

36. The system of claim 35, wherein said registration module provides for identifying applications to be governed by said policy.

37. The system of claim 32, wherein said module for detecting a second application detects an operating system call to open a communication channel to the first application.

38. The system of claim 32, wherein said module for detecting a second application detects a graphical device interface (GDI) message sent to the first application.

39. The system of claim 32, wherein said module for detecting a second application detects a local procedure call attempting to access the first application.

40. The system of claim 32, wherein said interprocess communication controller determines whether to permit the communication based, at least in part, upon evaluating parameters of the attempt made by the second application to communicate with the first application.

41. The system of claim 32, wherein said interprocess communication controller determines whether to permit the communication based upon obtaining user input as to whether to permit the second application to communicate with the first application.