SYSTEM, DEVICE AND METHOD FOR PROCESSING INTERLACED MULTIMODAL USER INPUT
1 Assignment
0 Petitions
Accused Products
Abstract
A device, method and system are provided for interpreting and executing operations based on multimodal input received at a computing device. The multimodal input can include one or more verbal and non-verbal inputs, such as a combination of speech and gesture inputs received substantially concurrently via suitable user interface means provided on the computing device. One or more target objects is identified from the non-verbal input, and text is recognized from the verbal input. An interaction object is generated using the recognized text and identified target objects, and thus comprises a natural language expression with embedded target objects. The interaction object is then processed to identify one or more operations to be executed.
-
Citations
57 Claims
-
1-35. -35. (canceled)
-
36. A method implemented at a computing device, the method comprising:
-
receiving verbal input using a verbal input interface of the computing device; receiving, concurrently with at least part of the verbal input, at least one secondary input using a non-verbal input interface of the computing device; identifying one or more target objects from the at least one secondary input; recognizing text from the received verbal input; generating an interaction object, the interaction object comprising a natural language expression having references to the one or more identified target objects embedded within the recognized text, the generating of the interaction object comprising identifying at least one attribute associated with each of the one or more identified target objects or at least one operation associated with each of the one or more identified target objects; processing the interaction object to identify at least one operation to be executed on at least one of the one or more identified target objects; and executing the operation on the at least one of the one or more identified target objects. - View Dependent Claims (37, 38, 39, 40, 41, 42, 43, 44, 45, 46)
-
-
47. A computing device, comprising:
-
at least one verbal input interface; at least one non-verbal input interface; at least one processor in communication with the at least one verbal input interface and the at least one non-verbal input interface, the at least one processor being configured to; receive verbal input using the verbal input interface; receive, concurrently with at least part of the verbal input, at least one secondary input using the at least one non-verbal input interface; identify one or more target objects from the at least one secondary input; recognize text from the received verbal input; generate an interaction object, the interaction object comprising a natural language expression having references to the one or more identified target objects embedded within the recognized text, the generation of the interaction object comprising identification of at least one attribute associated with each of the one or more identified target objects or at least one operation associated with each of the one or more identified target objects; process the interaction object to identify at least one operation to be executed on at least one of the one or more identified target objects; and execute the operation on the at least one of the one or more identified target objects. - View Dependent Claims (48, 49, 50, 51, 52, 53, 54, 55, 56, 57)
-
Specification