Natural interactive user interface using artificial intelligence and freeform input
First Claim
1. A device comprising:
- an operating system executed by a hardware processor,a graphical user interface component, when executing on the operating system, configured to detect a first user input including both handwriting input and non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input;
a handwriting recognition component, when executing on the operating system, configured to perform handwriting recognition on the handwriting input to transform the handwriting input into a textual input;
a first machine learning model, when executing on the operating system, trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action;
a second machine learning model, when executing on the operating system, trained to select a service from a plurality of services based on the textual input and a selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network; and
the graphical user interface component configured to;
display a selection of an action in the set of possible actions output by the first machine learning model having a highest probability score;
input a combination of the textual input and the selected action into the second machine learning model;
transform the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and
send the native request to the selected service for handling by the selected service.
1 Assignment
0 Petitions
Accused Products
Abstract
In an example embodiment, first user input including handwriting input and non-alphanumeric symbolic input is detected. The non-alphanumeric symbolic input is input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions. A combination of the action having the highest probability score and textual input from the handwriting input is input into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services. The combination of the textual input and the selected action is transformed into a native request for the selected service based on the service model for the selected service.
-
Citations
20 Claims
-
1. A device comprising:
-
an operating system executed by a hardware processor, a graphical user interface component, when executing on the operating system, configured to detect a first user input including both handwriting input and non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input; a handwriting recognition component, when executing on the operating system, configured to perform handwriting recognition on the handwriting input to transform the handwriting input into a textual input; a first machine learning model, when executing on the operating system, trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action; a second machine learning model, when executing on the operating system, trained to select a service from a plurality of services based on the textual input and a selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network; and the graphical user interface component configured to; display a selection of an action in the set of possible actions output by the first machine learning model having a highest probability score; input a combination of the textual input and the selected action into the second machine learning model; transform the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and send the native request to the selected service for handling by the selected service. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method comprising:
-
detecting first user input, the first user input including both a handwriting input and a non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input; transforming the handwriting input into textual input; inputting the non-alphanumeric symbolic input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and to output a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action; selecting an action in the set of possible actions output by the first machine learning model having a highest probability score; inputting a combination of the textual input and the selected action into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network; transforming the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and sending the native request to the selected service for handling by the selected service. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. A non-transitory machine-readable storage medium comprising instructions which, when implemented by one or more machines, cause the one or more machines to perform operations comprising:
-
detecting first user input, the first user input including both handwriting input and non-alphanumeric symbolic input, the non-alphanumeric symbolic input referencing at least a portion of the handwriting input; transforming the handwriting input into textual input; inputting the non-alphanumeric symbolic input into a first machine learning model trained to output a set of possible actions corresponding to the non-alphanumeric symbolic input and a probability score assigned to each action in the set of possible actions, the probability score indicating a likelihood that a user intended to invoke the corresponding action; selecting an action in the set of possible actions output by the first machine learning model having a highest probability score; inputting a combination of the textual input and the selected action into a second machine learning model trained to select a service from a plurality of services based on the textual input and the selected action by referencing a service model corresponding to each service in the plurality of services, wherein the second machine learning model is a neural network; transforming the combination of the textual input and the selected action into a native request for the selected service based on a service model for the selected service, the native request capable of being processed by the selected service without transformation; and sending the native request to the selected service for handling by the selected service. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification