System and method for managing files in a distributed system using filtering

• US 6,772,346 B1
• Filed: 07/16/1999
• Issued: 08/03/2004
• Est. Priority Date: 07/16/1999
• Status: Expired due to Term

• Current Assignee: Trend Micro Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Pring, Edward John, White, Steve Richard, Chess, David Michael, Kephart, Jeffrey Owen, Morar, John Frederick
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): KLIMACH, PAULA W
• Application Number:
  US09/356,770
• Time in Patent Office: 1,845 Days
• Field of Search: 701/213, 701/214, 714/38, 713/201, 713/200, 713/154, 707/6, 707/7, 707/200, 707/204, 707/205, 709/224, 709/225
• US Class Current: 709/223
• CPC Class Codes: H04L 63/0227 Filtering policies mail message filtering H04L51/212
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

In a network-connected distributed system including nodes through which digital data flow, one or more of the nodes adapted to process the digital data, a method for efficiently managing the transmission of units of digital data from node to node, includes the steps of receiving, at one of the one or more nodes, units of digital data first transmitted by an originating node; filtering out sufficiently processed units of the digital data based on filtering information; transmitting, to the originating node and/or other nodes, filtered results relating to the sufficiently processed units; queuing, for processing at other nodes, unfiltered units of the digital data which are not filtered out; and updating the filtering information according to results of automatic processing performed in and received from the one of the one or more nodes and/or other nodes in the system.

Claims

1. In a network-connected distributed system comprising a plurality of nodes through which digital data flow, at least one of the nodes adapted to process the digital data,

a method to efficiently manage the transmission of units of digital data from an originating node towards an analysis center node such that the analysis center node receives only one instance of at least one unit of digital data that requires analysis by the analysis center node, the method comprising:

receiving, at a node coupled between the originating node and the analysis center node, at least one unit of digital data first transmitted by the originating node;

filtering out, as filtered units of the digital data, any received units of the digital data that are found to meet at least one criterion of filtering information;

transmitting, to at least one of the originating node and other nodes, filtered results relating to the filtered units;

queuing, for processing at the at least one of the analysis center node and other nodes, unfiltered units of the digital data which are not filtered out; and

updating the filtering information in accordance with at least one of results of automatic processing performed by the node and results of automatic processing performed by and received from another node in the system.

2. The method of claim 1 wherein the distributed system includes nodes for the reporting and analysis of incorrect or buggy software, the units of digital data comprise files, and transmitting comprises returning updated information on bugs and fixes to at least one of the originating node and another node.

3. The method of claim 1 wherein the distributed system comprises a system for the analysis of geographically-based data, the units of digital data comprise requests for information about a particular geographical area, and transmitting comprises returning updated information on areas which have already been analyzed in response to prior queries to at least one of the originating node and another node.

4. The method of claim 1 wherein the distributed system comprises a system for the computation of integrals, and the units of digital data comprise queries of formulae to be integrated.

5. The method of claim 1 wherein the units of digital data comprise at least one of queries and files.

6. The method of claim 5 wherein the distributed system comprises a computer protection system, the units of digital data comprise at least one of files and checksums of files which are suspected to contain malicious code and transmitting comprises returning updated protection information to at least one of the originating node and the other nodes.

7. The method of claim 6 wherein the malicious code comprises at least one of computer viruses, worms and Trojan Horses.

8. The method of claim 6 wherein filtering comprises:

determining whether a file is identical to a known non-malicious file, and

in response to determining that the file is identical to a known malicious file, identifying the file as a filtered file.

9. The method of claim 8 wherein updating comprises:

receiving, from other nodes in the system, modification detection codes of files that have been determined to be non-malicious, and

adding the modification detection codes to the filtering information.

10. The method of claim 6 wherein filtering comprises:

determining whether a file cannot contain malicious code because it does not contain any code at all, and

in response to determining that the file does not contain any code at all, identifying the file as a filtered file.

11. The method of claim 6 wherein filtering comprises:

determining whether a file cannot contain malicious code because it does not contain enough code to constitute a smallest anticipated unit of malicious code; and

in response to determining that the file does not contain enough code to constitute the smallest anticipated unit of malicious code, identifying the file as a filtered file.

12. The method of claim 6 wherein filtering comprises:

determining whether a file contains known malicious code that is correctly handled by an existing protection definition, and

in response to determining that the file contains a known malicious code that is correctly handled by an existing protection definition, identifying the file as a filtered file.

13. The method of claim 12 wherein updating comprises:

receiving, from other nodes, protection definitions for malicious code that has been analyzed, and

adding the definitions to the filtering information.

14. The method of claim 1 wherein updating comprises re-executing filtering to apply the updated filtering information to the queued units of the digital data.

15. The method of claim 5 wherein the units of digital data comprise queries including a database version of the originating node and a request for an updated version, if available, wherein filtering comprises of determining whether a node has a more recent database version and wherein updating comprises updating originating filtering information of at least one of the originating node and other nodes of the system that are likely to have older versions.

16. The method of claim 15 wherein the database version corresponds to the filtering information.

17. The method of claim 5 wherein the units of digital data comprise queries including a database version of the originating node and a request for an updated version, if available, and wherein updating comprises of updating the originating filtering information of at least one of the originating node and other nodes of the system that are likely to have older versions.

18. The method of claim 5 wherein the distributed system comprises a computer protection system, the units of digital data comprise samples of spam and transmitting comprises returning updated protection information to at least one of the originating node and other nodes.

19. The method of claim 5 wherein the distributed system comprises a computer protection system, the units of digital data comprise samples of undesirable textual messages and transmitting comprises returning updated protection information to at least one of the originating node and other nodes.

20. In a network-connected distributed computer protection system comprising a plurality of nodes through which digital data flow, at least one of the nodes adapted to process the digital data,

a method to efficiently manage the transmission of suspect files from an originating node to a second node, the method comprising:

receiving, at the second node, a checksum of a suspect file transmitted by the originating node;

if a checksum match is found based on filtering information, identifying the suspect file as being a processed file;

else causing the receiving of the suspect file, at the second node;

filtering out, as filtered suspect files, any received suspect files that are found to meet at least one criterion of the filtering information;

transmitting, to at least one of the originating node and other nodes, filtered results relating to the filtered suspect files;

queuing, for processing at other nodes, unfiltered files which are not filtered out; and

updating the filtering information in accordance with at least one of results of automatic processing performed by the node and results of automatic processing performed by and received from another node in the system.

21. A system for efficiently managing the transmission of units of digital data from node to node in a distributed network comprising a plurality of nodes, said plurality of nodes comprising an analysis center node at least one originating node for transmitting units of digital data towards the analysis center node, and at least one node coupled between the at least one originating node and the analysis center node, the at least one node coupled between coupled between the at least one originating node and the analysis center node comprising:

a filter adapted to filter out units of the digital data received from the at least one originating node based on filtering information so that the filtered out units of digital data are not transmitted towards the analysis center node;

the filtering information being updatable according to results of automatic processing performed in and received from one of the plurality of nodes in the system.

22. In a network-connected distributed system comprising a plurality of nodes through which digital data flow, at least one of the nodes adapted to process the digital data,

a method to efficiently manage the transmission of units of digital data from an originating node towards an analysis center node such that the analysis center node receives only one instance of at least one unit of digital data that requires analysis by the analysis center node, the method comprising:

receiving, at a node coupled between the originating node and the analysis center node, at least one unit of digital data first transmitted by the originating node;

filtering out, as filtered units of the digital data, any received units of the digital data that are found to meet at least one criterion of filtering information;

transmitting, to at least one of the originating node and other nodes, filtered results relating to the filtered units;

queuing, for processing at the at least one of the analysis center node and other nodes, unfiltered units of the digital data which are not filtered out;

prioritizing the unfiltered units of digital data for transmission to a next node based on prioritizing information; and

updating the filtering information and the prioritizing information in accordance with at least one of results of automatic processing performed by the node and results of automatic processing performed by and received from another node in the system.

23. The method of claim 22 wherein updating comprises re-executing at least one of the filtering step and the prioritizing step to apply at least one of the updated filtering and prioritizing information to the queued units of the digital data.

24. The method of claim 22 wherein the units of digital data comprise at least one of queries and files.

25. A system for efficiently managing the transmission of units of digital data from node to node in a distributed network comprising a plurality of nodes, said plurality of nodes comprising an analysis center node, at least one originating node for transmitting units of digital data towards the analysis center node, and at least one node coupled between the at least one originating node and the analysis center node, the at least one node coupled between coupled between the at least one originating node and the analysis center node comprising:

a filter adapted to filter out units of the digital data received from the at least one originating node based on filtering information so that the filtered out units of digital data are not transmitted towards the analysis center node, the filtering information being updatable according to results of automatic processing performed in and received from one of the plurality of nodes in the system; and

a prioritizer adapted to prioritize units of the digital data queued for transmission to another node based on prioritizing information, the prioritizing information being updatable according to results of processing performed in and received from one of the plurality of nodes in the system.

26. A system for efficiently managing the transmission of units of digital data from an originating node towards an analysis center node via at least one intermediate node such that the analysis center node receives only one instance of at least one unit of digital data that requires analysis by the analysis center node, the system comprising a network-connected distributed computer protection system, the system further comprising:

means for receiving, at said at least one intermediate node, at least one unit of digital data first transmitted by the originating node;

means for filtering out, as filtered units of the digital data, any received units of the digital data that are found to meet at least one criterion of filtering information;

means for transmitting, to at least one of the originating node and other nodes, filtered results relating to the filtered units;

means for queuing, for processing at the at least one of the analysis center node and other nodes, unfiltered units of the digital data which are not filtered out; and

means for updating the filtering information in accordance with at least one of results of automatic processing performed by the node and results of automatic processing performed by and received from another node in the system.

27. The system of claim 26 wherein the nodes include nodes for the reporting and analysis of incorrect or buggy software, the units of digital data comprise files, and the transmitting means include means for returning updated information on bugs and fixes to at least one of the originating node and another node.

28. The system of claim 26 wherein the distributed system is adapted for the analysis of geographically-based data, the units of digital data comprise requests for information about a particular geographical area, and the transmitting means include means for returning updated information on areas which have already been analyzed in response to prior queries to at least one of the originating node and another node.

29. The system of claim 26 wherein the distributed system is adapted for the computation of integrals, and the units of digital data comprise queries of formulae to be integrated.

30. The system of claim 26 wherein the units of digital data comprise at least one of queries and files.

31. The system of claim 30 wherein the units of digital data comprise at least one of files and checksums of files which are suspected to contain malicious code and the transmitting means include means for returning updated protection information to at least one of the originating node and the other nodes.

32. The system of claim 31 wherein the malicious code comprises at least one of computer viruses, worms and Trojan Horses.

33. The system of claim 31 wherein the filtering means comprise:

means for determining whether a file is identical to a known non-malicious file, and

means, responsive to said determining means determining that the file is identical to a known malicious file, for identifying the file as a filtered file.

34. The system of claim 33 wherein the updating means comprise:

means for receiving, from other nodes in the system, modification detection codes of files that have been determined to be non-malicious, and

means for adding the modification detection codes to the filtering information.

35. The system of claim 31 wherein the filtering means comprise:

means for determining whether a file cannot contain malicious code because it does not contain any code at all, and

means responsive to said determining means determining that the file does not contain any code at all, for identifying the file as a filtered file.

36. The system of claim 31 wherein the filtering means comprise:

means for determining whether a file cannot contain malicious code because it does not contain enough code to constitute a smallest anticipated unit of malicious code; and

means, responsive to said determining means determining that the file does not contain enough code to constitute the smallest anticipated unit of malicious code, for identifying the file as a filtered file.

37. The system of claim 31 wherein the filtering means comprise:

means for determining whether a file contains known malicious code that is correctly handled by an existing protection definition, and

means, responsive to said determining means determining that the file contains a known malicious code that is correctly handled by an existing protection definition, for identifying the file as a filtered file.

38. The system of claim 37 wherein the updating means comprise:

means for receiving, from other nodes, protection definitions for malicious code that has been analyzed, and

means for adding the definitions to the filtering information.

39. The system of claim 26 wherein the updating means comprise means for re-filtering to apply the updated filtering information to the queued units of the digital data.

40. The system of claim 30 wherein the units of digital data comprise queries including a database version of the originating node and a request for an updated version, if available, wherein the filtering means comprise means for determining whether the one of the one or more nodes has a more recent database version and wherein the updating means comprise means for updating originating filtering information of at least one of the originating node and other nodes of the system that are likely to have older versions.

41. The system of claim 40 wherein the database version corresponds to the filtering information.

42. The system of claim 30 wherein the units of digital data comprise queries including a database version of the originating node and a request for an updated version, if available, and wherein the updating means comprise means for updating the originating filtering information of at least one of the originating node and other nodes of the system that are likely to have older versions.

43. The system of claim 41 wherein the database version corresponds to the filtering information.

44. The system of claim 30 wherein the units of digital data comprise samples of undesirable textual messages and the transmitting means include means for returning updated protection information to at least one of the originating node and other nodes.

45. A system for efficiently managing the transmission of units of digital data from node to node in a network-connected distributed computer protection system including a plurality of nodes through which digital data flow, at least one of the nodes adapted to process the digital data, the system comprising:

means for receiving, at one of the nodes, a checksum of a suspect file transmitted by an originating node;

if a checksum match is found based on filtering information, means for identifying the suspect file as a processed file;

else means for causing the receiving, at the one or more nodes, of the suspect file;

means for filtering out, as filtered suspect files, and received suspect files that are found to meet at least one criterion of the filtering information;

means for transmitting, to at least one of the originating node and other nodes, filtered results relating to the filtered suspect files;

means for queuing, for processing at other nodes, unfiltered files which are not filtered out; and

means for updating the filtering information in accordance with at least one of results of automatic processing performed by the node and results of automatic processing performed by and received from another node in the system.

46. A system for efficiently managing the transmission of units of digital data from an originating node towards an analysis center node via at least one intermediate node such that the analysis center node receives only one instance of at least one unit of digital data that requires analysis by the analysis center node, the system comprising a network-connected distributed computer system, the system further comprising:

means for receiving, at said at least one intermediate node, at least one unit of digital data first transmitted by the originating node;

means for filtering out, as filtered units of the digital data, any received units of the digital data that are found to meet at least one criterion of filtering information;

means for transmitting, to at least one of the originating node and other nodes, filtered results relating to the filtered units;

means for queuing, for processing at the at least one of the analysis center node and other nodes, unfiltered units of the digital data which are not filtered out;

means for prioritizing the unfiltered units of digital data for transmission to a next node based on prioritizing information; and

means for updating the filtering information and the prioritizing information in accordance with at least one of results of automatic processing performed by the node and results of automatic processing performed by and received from another node in the system.

47. The system of claim 46 wherein the updating means comprise means for at least one of re-filtering and re-prioritizing to apply at least one of the updated filtering and prioritizing information to the queued units of the digital data.

48. The method of claim 46 wherein the units of digital data comprise at least one queries and files.