System and method for assessing categorization rule selectivity

• US 9,501,742 B2
• Filed: 09/25/2014
• Issued: 11/22/2016
• Est. Priority Date: 12/05/2013
• Status: Active Grant

• Current Assignee: Lab A.O. Kaspersky
• Original Assignee: Lab A.O. Kaspersky
• Inventors: Antonov, Alexey E., Romanenko, Alexey M.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Figueroa, Kevin W
• Application Number:
  US14/496,848
• Publication Number:
  US 20150161523A1
• Time in Patent Office: 789 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

Assessment of selectivity of categorization rules. One or more categorization rules are applied to a set of un-categorized objects to produce a categorization result set representing assignment of objects the set into at least two categories. A selectivity score for the at least one categorization rule is obtained based on statistical information. The numerical selectivity score represents an estimation of accuracy of the at least one categorization rule, and is produced as a result of application of at least one trained selectivity determination algorithm, which is based on application of a plurality of specially-selected categorization rules to a set of pre-categorized training data, with the application of each one producing a uniform grouping of objects.

Claims

1. A system for assessing the selectivity of categorization rules, the system comprising:

a computer system including at least one processor, a non-transitory data storage medium interfaced with the at least one processor, and input/output facilities, the data storage medium containing instructions that, when executed by the at least one processor, implement;

a categorization rule application engine configured to apply at least one categorization rule to a set of un-categorized objects to produce a categorization result set representing assignment of objects of the set into at least two categories into which the objects of the set are divided when the categorization rule is applied, the categorization rule application engine further configured to gather statistical information relating to the categorization result set based on properties of objects assigned to each of the at least two categories, and including at least one rule-specific aggregating statistic characterizing the application of to the categorization rule to all of the objects and at least one categorization-specific statistic characterizing the objects of one of the at least two categories;

a selectivity determination engine configured to

assess a numerical selectivity score for the at least one categorization rule, the numerical selectivity score representing an estimation of selectivity accuracy of the at least one categorization rule to provide an evaluation of the at least one categorization rule, the numerical selectivity score being calculated by the application of at least one trained selectivity determination algorithm to the statistical information including the at least one rule-specific aggregating statistic representing information on the set of files belonging to each of the categories defined in the categorization rule, the application of the at least one trained selectivity determination algorithm to the statistical information including considering each of a plurality of parameters derived from the statistical information and in accordance with the at least one categorization rule,

and compare the selectivity score against a predefined selectivity threshold, wherein a selectivity score that exceeds the selectivity threshold is deemed highly selective; and

an algorithm training engine configured to produce each of the at least one trained selectivity determination algorithm based on application of a plurality of specially-selected categorization rules to a set of pre-categorized training data, wherein the application of each one of the specially-selected categorization rules to the set of training data produces at least one uniform grouping of objects in which the objects all meet a predefined similarity criterion, and wherein the trained selectivity determination algorithms are unrelated to the plurality of specially-selected categorization rules.

2. The system of claim 1, wherein the set of un-categorized objects consists of files.

3. The system of claim 1, wherein the categorization rules are malware detection rules.

4. The system of claim 1, further comprising:

a categorization rule creation engine implemented by the instructions, and configured to autonomously create categorization rules.

5. The system of claim 1, wherein the predefined similarity criterion is based on similarity of content of objects.

6. The system of claim 1, wherein the predefined similarity criterion is based on similarity of functionality of objects.

7. The system of claim 1, wherein the un-categorized objects reside on the computer system.

8. The system of claim 1, wherein the properties of objects assigned to each of the at least two categories on which the categorization result set is based include at least one set of properties selected from the group consisting of:

elapsed time since a prior application of a categorization rule, a quantity of objects categorized into a category by application of a categorization rule, a size of objects categorized into a category by application of a categorization rule, or any combination thereof.

9. The system of claim 1, wherein the properties of objects assigned to each of the at least two categories on which the categorization result set is based include at least one set of properties selected from the group consisting of:

unique compilers used to create objects categorized into a category by application of a categorization rule, unique packagers used to create objects categorized into a category by application of a categorization rule, or any combination thereof.

10. The system of claim 1, wherein the selectivity determination engine is configured to apply a plurality of different trained selectivity determination algorithms to produce a corresponding plurality of individual selectivity scores, and wherein the selectivity determination engine is further configured to aggregate the plurality of individual selectivity scores to produce a single total selectivity score for each categorization rule.

21. The system of claim 1, wherein the trained selectivity determination algorithm uses a decision tree, and wherein each of the plurality of parameters is a node on the decision tree.

11. A machine-implemented method for assessing the selectivity of categorization rules, the method comprising:

autonomously applying at least one categorization rule to a set of un-categorized objects to produce a categorization result set representing assignment of objects of the set into at least two categories into which the objects of the set are divided when the categorization rule is applied;

autonomously gathering statistical information relating to the categorization result set based on properties of objects assigned to each of the at least two categories, and including at least one rule-specific aggregating statistic characterizing the application of to the categorization rule to all of the objects and at least one categorization-specific statistic characterizing the objects of one of the at least two categories;

autonomously assessing a numerical selectivity score for the at least one categorization rule, the numerical selectivity score representing an estimation of selectivity accuracy of the at least one categorization rule to provide an evaluation of the at least one categorization rule, the numerical selectivity score being calculated by the application of at least one trained selectivity determination algorithm to the statistical information including the at least one rule-specific aggregating statistic representing information on the set of files belonging to each of the categories defined in the categorization rule, the application of the at least one trained selectivity determination algorithm to the statistical information including considering each of a plurality of parameters derived from the statistical information and in accordance with the at least one categorization rule;

autonomously producing each of the at least one trained selectivity determination algorithm based on application of a plurality of specially-selected categorization rules to a set of pre-categorized training data, wherein the application of each one of the specially-selected categorization rules to the set of training data produces at least one uniform grouping of objects in which the objects all meet a predefined similarity criterion, and wherein the trained selectivity determination algorithms are unrelated to the plurality of specially-selected categorization rules and comparing the selectivity score against a predefined selectivity threshold, wherein a selectivity score that exceeds the selectivity threshold is deemed highly selective.

12. The method of claim 11, wherein the set of un-categorized objects consists of files.

13. The method of claim 11, wherein the categorization rules are malware detection rules.

14. The method of claim 11, further comprising:

autonomously creating the categorization rules.

15. The method of claim 11, wherein the predefined similarity criterion is based on similarity of content of objects.

16. The method of claim 11, wherein the predefined similarity criterion is based on similarity of functionality of objects.

17. The method of claim 11, wherein the properties of objects assigned to each of the at least two categories on which the categorization result set is based include at least one set of properties selected from the group consisting of:

elapsed time since a prior application of a categorization rule, a quantity of objects categorized into a category by application of a categorization rule, a size of objects categorized into a category by application of a categorization rule, or any combination thereof.

18. The method of claim 11, wherein the properties of objects assigned to each of the at least two categories on which the categorization result set is based include at least one set of properties selected from the group consisting of:

unique compilers used to create objects categorized into a category by application of a categorization rule, unique packagers used to create objects categorized into a category by application of a categorization rule, or any combination thereof.

19. The method of claim 11, wherein a plurality of different trained selectivity determination algorithms are applied to produce a corresponding plurality of individual selectivity scores, and wherein the plurality of individual selectivity scores are aggregated to produce a single total selectivity score for each categorization rule.

22. The method of claim 11, wherein the trained selectivity determination algorithm uses a decision tree, and wherein each of the plurality of parameters is a node on the decision tree.

20. A system for assessing the selectivity of categorization rules, the method comprising:

means for applying at least one categorization rule to a set of un-categorized objects to produce a categorization result set representing assignment of objects of the set into at least two categories into which the objects of the set are divided when the categorization rule is applied and for gathering statistical information relating to the categorization result set based on properties of objects assigned to each of the at least two categories, and including at least one rule-specific aggregating statistic characterizing the application of to the categorization rule to all of the objects and at least one categorization-specific statistic characterizing the objects of one of the at least two categories;

means for assessing a numerical selectivity score for the at least one categorization rule, the numerical selectivity score representing an estimation of selectivity accuracy of the at least one categorization rule to provide an evaluation of the at least one categorization rule, the numerical selectivity score being calculated by the application of at least one trained selectivity determination algorithm to the statistical information including the at least one rule-specific aggregating statistic representing information on the set of files belonging to each of the categories defined in the categorization rule, the application of the at least one trained selectivity determination algorithm to the statistical information including considering each of a plurality of parameters derived from the statistical information and in accordance with the at least one categorization rule, and comparing the selectivity score against a predefined selectivity threshold, wherein a selectivity score that exceeds the selectivity threshold is deemed highly selective; and

means for producing each of the at least one trained selectivity determination algorithm based on application of a plurality of specially-selected categorization rules to a set of pre-categorized training data, wherein the application of each one of the specially-selected categorization rules to the set of training data produces at least one uniform grouping of objects in which the objects all meet a predefined similarity criterion, and wherein the trained selectivity determination algorithms are unrelated to the plurality of specially-selected categorization rules.