Natural interactive user interface using artificial intelligence and freeform input

• US 10,402,740 B2
• Filed: 07/29/2016
• Issued: 09/03/2019
• Est. Priority Date: 07/29/2016
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Hoetzer, Judith, Miseldine, Philip
• Primary Examiner(s): Paula, Cesar B
• Assistant Examiner(s): Nguyen, Luu-Phuong T
• Application Number:
  US15/223,991
• Publication Number:
  US 20180032505A1
• Time in Patent Office: 1,131 Days
• Field of Search: 715268
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/12 Use of codes for handling textual entities
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 40/08 Insurance

Abstract

In an example embodiment, first user input including handwriting input and non-alphanumeric symbolic input is detected. The non-alphanumeric symbolic input is input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions. A combination of the action having the highest probability score and textual input from the handwriting input is input into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services. The combination of the textual input and the selected action is transformed into a native request for the selected service based on the service model for the selected service.

Claims

1. A device comprising:

an operating system executed by a hardware processor,

a graphical user interface component, when executing on the operating system, configured to detect a first user input including both handwriting input and non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input;

a handwriting recognition component, when executing on the operating system, configured to perform handwriting recognition on the handwriting input to transform the handwriting input into a textual input;

a first machine learning model, when executing on the operating system, trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action;

a second machine learning model, when executing on the operating system, trained to select a service from a plurality of services based on the textual input and a selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network; and

the graphical user interface component configured to;

display a selection of an action in the set of possible actions output by the first machine learning model having a highest probability score;

input a combination of the textual input and the selected action into the second machine learning model;

transform the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and

send the native request to the selected service for handling by the selected service.

2. The system of claim 1, wherein the non-alphanumeric symbolic input includes gesture-based input.

3. The system of claim 1, wherein the non-alphanumeric symbolic input includes a drawing made by the user via the graphical user interface.

4. The system of claim 1, wherein the non-alphanumeric symbolic input includes circling an object rendered in the graphical user interface and connecting the circled object to the handwriting input.

5. The system of claim 1, wherein the second machine learning model is coupled to a plurality of different semantic models of services, each semantic model of a service defining how the corresponding service is used, its domain, and its construction, and containing one or more native requests for the corresponding service.

6. The system of claim 5, wherein each of the one or more native requests has a set of input parameters, and wherein the second machine learning model is further trained to output a selected native request for the selected service based on the set of input parameters for the selected native request.

7. The system of claim 1, wherein each of the plurality of services is located on a different data source.

8. A method comprising:

detecting first user input, the first user input including both a handwriting input and a non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input;

transforming the handwriting input into textual input;

inputting the non-alphanumeric symbolic input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and to output a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action;

selecting an action in the set of possible actions output by the first machine learning model having a highest probability score;

inputting a combination of the textual input and the selected action into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network;

transforming the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and

sending the native request to the selected service for handling by the selected service.

9. The method of claim 8, wherein the non-alphanumeric symbolic input includes gesture-based input.

10. The method of claim 8, wherein the non-alphanumeric symbolic input includes a drawing made by the user via a graphical user interface.

11. The method of claim 8, wherein the non-alphanumeric symbolic input includes circling an object rendered in a graphical user interface and connecting the circled object to the handwriting input.

12. The method of claim 8, wherein the second machine learning model is coupled to a plurality of different semantic models of services, each semantic model of a service defining how the corresponding service is used, its domain, and its construction, and containing one or more native requests for the corresponding service.

13. The method of claim 12, wherein each of the one or more native requests has a set of input parameters, and wherein the second machine learning model is further trained to output a selected native request for the selected service based on the set of input parameters for the selected native request.

14. The method of claim 8, wherein each of the plurality of services is located on a different data source.

15. A non-transitory machine-readable storage medium comprising instructions which, when implemented by one or more machines, cause the one or more machines to perform operations comprising:

detecting first user input, the first user input including both handwriting input and non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input;

transforming the handwriting input into textual input;

inputting the non-alphanumeric symbolic input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action;

selecting an action in the set of possible actions output by the first machine learning model having a highest probability score;

inputting a combination of the textual input and the selected action into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network;

transforming the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and

sending the native request to the selected service for handling by the selected service.

16. The non-transitory machine-readable storage medium of claim 15, wherein the non-alphanumeric symbolic input includes gesture-based input.

17. The non-transitory machine-readable storage medium of claim 15, wherein the non-alphanumeric symbolic input includes a drawing made by the user via a graphical user interface.

18. The non-transitory machine-readable storage medium of claim 15, wherein the non-alphanumeric symbolic input includes circling an object rendered in a graphical user interface and connecting the circled object to the handwriting input.

19. The non-transitory machine-readable storage medium of claim 15, wherein the second machine learning model is coupled to a plurality of different semantic models of services, each semantic model of a service defining how the corresponding service is used, its domain, and its construction, and containing one or more native requests for the corresponding service.

20. The non-transitory machine-readable storage medium of claim 19, wherein each of the one or more native requests has a set of input parameters, and wherein the second machine learning model is further trained to output a selected native request for the selected service based on the set of input parameters for the selected native request.