Example-based item classification

• US 10,146,863 B2
• Filed: 11/17/2014
• Issued: 12/04/2018
• Est. Priority Date: 11/17/2014
• Status: Active Grant

• Current Assignee: GOTO Group Incorporated (LogMeIn, Inc.)
• Original Assignee: GetGo, Inc. (LogMeIn, Inc.)
• Inventors: Thapliyal, Ashish V., Lock, Anne Marie, Thapliyal, Elizabeth, Kasper, Ryan W., von Imhof, Stefan Alexander
• Primary Examiner(s): Saeed, Usmaan
• Assistant Examiner(s): Kim, Paul
• Application Number:
  US14/543,230
• Publication Number:
  US 20160140222A1
• Time in Patent Office: 1,478 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]

Abstract

Item classification rules are created based on examples selected by a user, such as by selecting a subset of emails, and the rule is used across a larger set of items to obtain automatic classification of similar items according to the rule. Based on an analysis, a candidate classification rule is generated identifying text-based features shared among the items of the subset. The user can review the candidate rule as well as a resultant subset of items generated by the rule, and either accept the candidate rule or make an adjustment to the examples and then perform one or more iterations of the analysis to refine the rule. Adjustments can be made by removing items incorrectly included in a resultant subset and/or adding items incorrectly excluded from a resultant subset, and using the adjusted subset in a next iteration.

Claims

1. A method of operating a computerized device according to an electronic mail application program to selectively classify items of a set of electronic mail messages and display classified electronic mail messages to a user, comprising:

receiving a user selection of an example subset of the set of electronic mail messages;

performing an analysis on the example subset to find one or more shared text-based features that are shared across all items of the example subset, and based on the analysis generating a candidate classification rule identifying the shared text-based features, the analysis including multiple steps performed in sequence for respective distinct fields of the electronic mail messages, wherein in a first pass the analysis stops at a first step at which text content is found that is shared across all items of the example subset, and wherein the analysis for a next iteration continues past the first step to a second step at which text content is found that is shared across all items of the example subset;

applying the candidate classification rule to the set of electronic mail messages to identify a resultant subset of the electronic mail messages satisfying the candidate classification rule, the resultant subset being a superset of the example subset;

displaying the resultant subset to the user and receiving user input indicating, based on user review of the resultant subset, whether the candidate classification rule is accepted;

if the user input indicates that the candidate classification rule is accepted, then finalizing the candidate classification rule into a final classification rule, and otherwise repeating the above steps one or more times for respective adjusted example subsets of the set of electronic mail messages until a respective adjusted candidate classification rule is accepted and finalized into the final classification rule, the adjusted example subset for a given repetition being formed by the user identifying a false match and/or a false non-match with respect to the resultant subset identified in the preceding repetition; and

automatically applying the final classification rule in subsequent operation of the electronic mail application program to identify electronic mail messages satisfying the final classification rule and to display the identified electronic mail messages to the user in a manner reflecting their identification as satisfying the final classification rule.

2. A method according to claim 1, wherein receiving the user selection includes operating a graphical user interface to respond to a set of graphical user selection actions on a presentation of the set of electronic mail messages to identify the items of the example subset stored in an item store.

3. A method according to claim 1, wherein the field of the electronic mail messages include one or more structured fields and one or more unstructured fields, and wherein the analysis includes multiple steps performed in sequence beginning with text content of the structured fields of the electronic mail messages and proceeding to text content of the unstructured fields of the electronic mail messages, the analysis stopping at a first step at which text content is found that is shared across all electronic mail messages of the example subset.

4. A method according to claim 3, wherein the structured fields include one or more sender fields and one or more recipient fields, and the unstructured fields include a message body containing user-created message text.

5. A method according to claim 3, wherein the unstructured fields include an attachment containing arbitrary text.

6. A method according to claim 1, wherein receiving the user input includes receiving an identification of an electronic mail message omitted from the resultant subset, and further including adding the omitted electronic mail message to the example subset to form the adjusted example subset for a next repetition.

7. A method according to claim 1, wherein receiving the user input includes receiving an identification of an electronic mail message incorrectly included the resultant subset, and further including removing the incorrectly included electronic mail message from the example subset to form the adjusted example subset for a next repetition.

8. A method according to claim 1, further including, upon completing one or more of the repetitions with respective adjusted example subsets without the user accepting the respective adjusted candidate rule, incorporating a negative condition and performing additional repetitions using both a respective adjusted example subset and the negative condition, the negative condition specifying a text-based feature contained in the incorrectly included electronic mail message and not shared with the electronic mail messages of the adjusted example subset, the negative condition used to exclude electronic mail messages containing the text-based feature from the resultant subset of one or more of the additional repetitions.

9. A method according to claim 1, wherein displaying the resultant subset includes displaying first electronic mail messages for which there is a first confidence level about proper classification, and omitting second electronic mail messages for which there is a second, higher confidence level about proper classification.

10. A method according to claim 1, further including displaying to the user one or more electronic mail messages omitted from the resultant subset to enable the user to add the omitted electronic mail messages to the example subset to form the adjusted example subset for a next repetition.

19. A method according to claim 1, further including:

maintaining a store of finalized and actively used classification rules regularly applied during operation of the electronic mail application program; and

adding the final classification rule to the store and retrieving the final classification rule from the store when applying it in subsequent operation.

20. A method according to claim 1, wherein displaying the identified electronic mail messages to the user includes interfacing with a graphical user interface of the computer to generate one or more of icons, shadings, and foldering as graphical indications of classification of the electronic mail messages.

11. A non-transitory computer-readable medium storing computer program instructions of an electronic mail application program, the instructions being executable by a computer to cause the computer to selectively classify items of a set of electronic mail messages and display classified electronic mail messages to a user, the method including:

receiving a user selection of an example subset of the set of electronic mail messages;

performing an analysis on the example subset to find one or more shared text-based features that are shared across all items of the example subset, and based on the analysis generating a candidate classification rule identifying the shared text-based features, the analysis including multiple steps performed in sequence for respective distinct fields of the electronic mail messages, wherein in a first pass the analysis stops at a first step at which text content is found that is shared across all items of the example subset, and wherein the analysis for a next iteration continues past the first step to a second step at which text content is found that is shared across all items of the example subset;

applying the candidate classification rule to the set of electronic mail messages to identify a resultant subset of the electronic mail messages satisfying the candidate classification rule, the resultant subset being a superset of the example subset;

displaying the resultant subset to the user and receiving user input indicating, based on user review of the resultant subset, whether the candidate classification rule is accepted;

if the user input indicates that the candidate classification rule is accepted, then finalizing the candidate classification rule into a final classification rule, and otherwise repeating the above steps one or more times for respective adjusted example subsets of the set of electronic mail messages until a respective adjusted candidate classification rule is accepted and finalized into the final classification rule, the adjusted example subset for a given repetition being formed by the user identifying a false match and/or a false non-match with respect to the resultant subset identified in the preceding repetition; and

automatically applying the final classification rule in subsequent operation of the electronic mail application program to identify electronic mail messages satisfying the final classification rule and to display the identified electronic mail messages to the user in a manner reflecting their identification as satisfying the final classification rule.

12. A non-transitory computer-readable medium according to claim 11, wherein each of the electronic mail messages includes one or more structured fields and one or more unstructured fields, and wherein the analysis includes multiple steps performed in sequence beginning with text content of the structured fields of the electronic mail messages and proceeding to text content of the unstructured fields of the electronic mail messages, the analysis stopping at a first step at which text content is found that is shared across all electronic mail messages of the example subset.

13. A non-transitory computer-readable medium according to claim 12, wherein the structured fields include one or more sender fields and one or more recipient fields, and the unstructured fields include a message body containing user-created message text.

14. A non-transitory computer-readable medium according to claim 11, wherein receiving the user input includes receiving an identification of an electronic mail message omitted from the resultant subset, and further including adding the omitted electronic mail message to the example subset to form the adjusted example subset for a next repetition.

15. A non-transitory computer-readable medium according to claim 11, wherein receiving the user input includes receiving an identification of an electronic mail message incorrectly included the resultant subset, and further including removing the incorrectly included electronic mail message from the example subset to form the adjusted example subset for a next repetition.

16. A non-transitory computer-readable medium according to claim 11, wherein the method performed by the computer further includes, upon completing one or more of the repetitions with respective adjusted example subsets without the user accepting the respective adjusted candidate rule, incorporating a negative condition and performing additional repetitions using both a respective adjusted example subset and the negative condition, the negative condition specifying a text-based feature contained in the incorrectly included electronic mail message and not shared with the electronic mail messages of the adjusted example subset, the negative condition used to exclude electronic mail messages containing the text-based feature from the resultant subset of one or more of the additional repetitions.

17. A non-transitory computer-readable medium according to claim 11, wherein displaying the resultant subset includes displaying first electronic mail messages for which there is a first confidence level about proper classification, and omitting second electronic mail messages for which there is a second, higher confidence level about proper classification.

18. A non-transitory computer-readable medium according to claim 11, wherein the method performed by the computer further includes displaying to the user one or more electronic mail messages omitted from the resultant subset to enable the user to add the omitted electronic mail message to the example subset to form the adjusted example subset for a next repetition.