EXAMPLE-BASED ITEM CLASSIFICATION

• US 20160140222A1
• Filed: 11/17/2014
• Published: 05/19/2016
• Est. Priority Date: 11/17/2014
• Status: Active Grant

• Current Assignee: GOTO Group Incorporated (LogMeIn, Inc.)
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Lock, Anne Marie, Kasper, Ryan W., von Imhof, Stefan Alexander, Thapliyal, Ashish V., Thapliyal, Elizabeth
• Granted Patent: US 10,146,863 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]

Abstract

Item classification rules are created based on examples selected by a user, such as by selecting a subset of emails, and the rule is used across a larger set of items to obtain automatic classification of similar items according to the rule. Based on an analysis, a candidate classification rule is generated identifying text-based features shared among the items of the subset. The user can review the candidate rule as well as a resultant subset of items generated by the rule, and either accept the candidate rule or make an adjustment to the examples and then perform one or more iterations of the analysis to refine the rule. Adjustments can be made by removing items incorrectly included in a resultant subset and/or adding items incorrectly excluded from a resultant subset, and using the adjusted subset in a next iteration.

Claims

1. A method of operating a computerized device to generate a classification rule for use in an application program to selectively classify items of a set of items and display classified items to a user, comprising:

receiving a user selection of an example subset of the set of items;

performing an analysis on the example subset to find one or more shared text-based features that are shared across all items of the example subset, and based on the analysis generating a candidate classification rule identifying the shared text-based features;

applying the candidate classification rule to the set of items to identify a resultant subset of the items satisfying the candidate classification rule, the resultant subset being a superset of the example subset;

displaying the resultant subset to the user and receiving user input indicating, based on user review of the resultant subset, whether the candidate classification rule is accepted; and

if the user input indicates that the candidate classification rule is accepted, then finalizing the candidate classification rule into a final classification rule for future use by the application program in classifying items, and otherwise repeating the above steps one or more times for respective adjusted example subsets of the set of items until a respective adjusted candidate classification rule is accepted and finalized into the final classification rule, the adjusted example subset for a given repetition being formed by the user identifying a false match and/or a false non-match with respect to the resultant subset identified in the preceding repetition.

2. A method according to claim 1, wherein receiving the user selection includes operating a graphical user interface to respond to a set of graphical user selection actions on a presentation of the set of items to identify the items of the example subset stored in an item store.

3. A method according to claim 1, wherein each of the items includes one or more structured fields and one or more unstructured fields, and wherein the analysis includes multiple steps performed in sequence beginning with text content of the structured fields of the items and proceeding to text content of the unstructured fields of the items, the analysis stopping at a first step at which text content is found that is shared across all items of the example subset.

4. A method according to claim 3, wherein the items are electronic mail messages, the structured fields include one or more sender fields and one or more recipient fields, and the unstructured fields include a message body containing user-created message text.

5. A method according to claim 3, wherein the items are electronic mail messages, and the unstructured fields include an attachment containing arbitrary text.

6. A method according to claim 1, wherein the items are electronic mail messages.

7. A method according to claim 1, wherein receiving the user input includes receiving an identification of an item omitted from the resultant subset, and further including adding the omitted item to the example subset to form the adjusted example subset for a next repetition.

8. A method according to claim 1, wherein receiving the user input includes receiving an identification of an item incorrectly included the resultant subset, and further including removing the incorrectly included item from the example subset to form the adjusted example subset for a next repetition.

9. A method according to claim 8, wherein the analysis includes multiple steps performed in sequence for respective distinct fields of the items, and in a first pass the analysis stops at a first step at which text content is found that is shared across all items of the example subset, and wherein the analysis for the next iteration continues past the first step to a second step at which text content is found that is shared across all items of the example subset.

10. A method according to claim 9, further including, upon completing one or more of the repetitions with respective adjusted example subsets without the user accepting the respective adjusted candidate rule, incorporating a negative condition and performing additional repetitions using both a respective adjusted example subset and the negative condition, the negative condition specifying a text-based feature contained in the incorrectly included item and not shared with the items of the adjusted example subset, the negative condition used to exclude items containing the text-based feature from the resultant subset of one or more of the additional repetitions.

11. A method according to claim 1, wherein displaying the resultant subset includes displaying messages along with information expressing respective levels of certainty about proper classification of the messages.

12. A method according to claim 1, wherein performing the analysis includes using natural language processing procedures.

13. A method according to claim 1, further including displaying to the user one or more items omitted from the resultant subset to enable the user to add the omitted item to the example subset to form the adjusted example subset for a next repetition.

14. A non-transitory computer-readable medium storing computer program instructions, the instructions being executable by a computer to cause the computer to perform a method of generating a classification rule for use in an application program to selectively classify items of a set of items and display classified items to a user, the method including:

receiving a user selection of an example subset of the set of items;

performing an analysis on the example subset to find one or more shared text-based features that are shared across all items of the example subset, and based on the analysis generating a candidate classification rule identifying the shared text-based features;

applying the candidate classification rule to the set of items to identify a resultant subset of the items satisfying the candidate classification rule, the resultant subset being a superset of the example subset;

displaying the resultant subset to the user and receiving user input indicating, based on user review of the resultant subset, whether the candidate classification rule is accepted; and

if the user input indicates that the candidate classification rule is accepted, then finalizing the candidate classification rule into a final classification rule for future use by the application program in classifying items, and otherwise repeating the above steps one or more times for respective adjusted example subsets of the set of items until a respective adjusted candidate classification rule is accepted and finalized into the final classification rule, the adjusted example subset for a given repetition being formed by the user identifying a false match and/or a false non-match with respect to the resultant subset identified in the preceding repetition.

15. A non-transitory computer-readable medium according to claim 14, wherein each of the items includes one or more structured fields and one or more unstructured fields, and wherein the analysis includes multiple steps performed in sequence beginning with text content of the structured fields of the items and proceeding to text content of the unstructured fields of the items, the analysis stopping at a first step at which text content is found that is shared across all items of the example subset.

16. A non-transitory computer-readable medium according to claim 15, wherein the items are electronic mail messages, the structured fields include one or more sender fields and one or more recipient fields, and the unstructured fields include a message body containing user-created message text.

17. A non-transitory computer-readable medium according to claim 14, wherein receiving the user input includes receiving an identification of an item omitted from the resultant subset, and further including adding the omitted item to the example subset to form the adjusted example subset for a next repetition.

18. A non-transitory computer-readable medium according to claim 14, wherein receiving the user input includes receiving an identification of an item incorrectly included the resultant subset, and further including removing the incorrectly included item from the example subset to form the adjusted example subset for a next repetition.

19. A non-transitory computer-readable medium according to claim 18, wherein the analysis includes multiple steps performed in sequence for respective distinct fields of the items, and in a first pass the analysis stops at a first step at which text content is found that is shared across all items of the example subset, and wherein the analysis for the next iteration continues past the first step to a second step at which text content is found that is shared across all items of the example subset.

20. A non-transitory computer-readable medium according to claim 19, wherein the method performed by the computer further includes, upon completing one or more of the repetitions with respective adjusted example subsets without the user accepting the respective adjusted candidate rule, incorporating a negative condition and performing additional repetitions using both a respective adjusted example subset and the negative condition, the negative condition specifying a text-based feature contained in the incorrectly included item and not shared with the items of the adjusted example subset, the negative condition used to exclude items containing the text-based feature from the resultant subset of one or more of the additional repetitions.