Machine learning visualization

• US 10,732,789 B1
• Filed: 03/12/2019
• Issued: 08/04/2020
• Est. Priority Date: 03/12/2019
• Status: Active Grant

• Current Assignee: Bottomline Technologies Incorporated
• Original Assignee: Bottomline Technologies Incorporated
• Inventors: Marcinelli, Michael, Kidani, Flora, Canneto, John, Baron, Anne, Hewitt, Jonathan, Cashman, William
• Primary Examiner(s): Rodriguez, Daniel
• Application Number:
  US16/299,227
• Time in Patent Office: 511 Days
• Field of Search: 715764
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/109 Font handling; Temporal or kinetic typography
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A unique user interface for improving machine learning algorithms is described herein. The user interface comprises an icon with multiple visual indicators displaying the machine learning confidence score. When a mouse hovers over the icon, a set of icons are displayed to accept the teaching user'"'"'s input. In addition, the words that drove the machine learning confidence score are highlighted with formatting so that the teaching user can understand what drove the machine learning confidence score.

Claims

1. A special purpose computer implemented method of visualizing a machine learning confidence score, the method comprising:

processing a textual description through a machine learning model to derive the machine learning confidence score, wherein the machine learning model uses natural language processing to convert the textual description into word stems that are used by the machine learning model to calculate the machine learning confidence score, and then searches the textual description for the word stems that comprised a highest impact in the machine learning confidence score, adding formatting instructions to bold or increase a font size at least one word associated with the word stems that comprised the highest impact on the machine learning confidence score;

displaying on a display screen the textual description with the formatting instructions to indicate reasoning used by the machine learning model in a determination of the machine learning confidence score, wherein the textual description with the formatting instructions is only displayed on the display screen when a mouse location is over a variable icon that varies depending on a magnitude of the machine learning confidence score; and

accepting input from a user correcting the machine learning confidence score, said input used to teach the machine learning model.

2. The method of claim 1 wherein the textual description is from a legal invoice.

3. The method of claim 1 wherein the textual description is from an expert witness invoice.

4. The method of claim 1 further comprising displaying the variable icon on the display screen to graphically indicate the machine learning confidence score.

5. The method of claim 4 wherein the variable icon displays a different number of items depending upon the magnitude of the machine learning confidence score.

6. The method of claim 5 wherein the variable icon displays three rays off of a lightbulb based on one range of the machine learning confidence score, and one ray off of the lightbulb based on another range of the machine learning confidence score.

7. The method of claim 4 wherein the variable icon changes once the input from the user is accepted.

8. The method of claim 1 wherein the user input modifies the machine learning model in real time.

9. An apparatus for the visualization of a machine learning confidence score, the apparatus comprising:

a display screen;

a special purpose computer electrically connected to the display screen;

a large capacity data storage facility with a machine learning training data set;

a user interface display module operating on the special purpose computer and displaying an indication of the machine learning confidence score on the display screen;

a natural language processing module operating on the special purpose computer and interfacing with the user interface display module to convert a textual description into a table of word stems;

a machine learning module operating on the special purpose computer and interfacing with the natural language processing module to convert the table of the word stems into the machine learning confidence score using a machine learning model built using the machine learning training data set;

a scan back module operating on the special purpose computer and interfacing to the machine learning module to insert formatting instructions to bold or increase a font size into the textual description using the table of stems to format at least one word in the textual description that impacted the machine learning confidence score; and

a user interface input module operating on the special purpose computer and interfacing with the scan back module to display the textual description with the formatting instructions and to accept user input regarding the machine learning confidence score and to add the user input to the machine learning training data set, wherein the textual description with the formatting instructions is only displayed on the display screen when a mouse location is over a variable icon that varies depending on a magnitude of the machine learning confidence score.

10. The apparatus of claim 9 wherein the textual description is from a legal invoice.

11. The apparatus of claim 9 wherein the textual description is from an independent adjustor invoice.

12. The apparatus of claim 9 wherein the indication of the machine learning confidence score on the display screen is in a form of the variable icon that graphically varies depending on the magnitude of the machine learning confidence score.

13. The apparatus of claim 12 wherein the variable icon displays a different number of items depending upon the magnitude of the machine learning confidence score.

14. The apparatus of claim 13 wherein the variable icon displays three rays off of a lightbulb based on one range of the machine learning confidence score, and one ray off of the lightbulb based on another range of the machine learning confidence score.

15. The apparatus of claim 9 wherein the user input modifies the machine learning model in real time.

16. A system for a visualization of a machine learning confidence score, the system comprising:

a display screen on a personal computing device;

a special purpose computer electrically connected to the personal computing device through a network;

a large capacity data storage facility electrically connected to the special purpose computer, the large capacity data storage facility containing a machine learning training data set;

a user interface display module operating on the personal computing device and displaying the machine learning confidence score on the display screen in a form of a variable icon that varies depending on a magnitude of the machine learning confidence score;

a natural language processing module operating on the special purpose computer to convert a textual description into a table of word stems;

a machine learning module operating on the special purpose computer and interfacing with the natural language processing module to convert the table of the word stems into the machine learning confidence score using a machine learning model built using the machine learning training data set;

a scan back module operating on the special purpose computer and interfacing to the machine learning module to insert formatting instructions to bold or increase a font size into the textual description using the machine learning confidence score and the table of word stems to format at least one word in the textual description that impacted the machine learning confidence score; and

a user interface input module operating on the personal computing device and interfacing with the scan back module through the network to display the textual description with the formatting instructions and to accept user input regarding the machine learning confidence score and sending the user input to the special purpose computer to add the user input to the machine learning training data set, wherein the textual description with the formatting instructions is only displayed on the display screen when a mouse location is over the variable icon.

17. The system of claim 16 wherein the textual description is from a legal invoice.

18. The system of claim 16 wherein the variable icon displays a different number of items depending upon the magnitude of the machine learning confidence score.

19. The system of claim 18 wherein the variable icon displays three rays off of a lightbulb based on one range of the machine learning confidence score, and one ray off of the lightbulb based on another range of the machine learning confidence score.

20. The system of claim 16 wherein the variable icon changes once the user input from the user is accepted.