System, device and method for processing interlaced multimodal user input

• US 9,601,113 B2
• Filed: 05/15/2013
• Issued: 03/21/2017
• Est. Priority Date: 05/16/2012
• Status: Active Grant

• Current Assignee: Xtreme Interactions Inc.
• Original Assignee: Xtreme Interactions Inc.
• Inventors: Anandarajah, Joe
• Primary Examiner(s): Augustin, Marcellus
• Application Number:
  US14/241,399
• Publication Number:
  US 20150019227A1
• Time in Patent Office: 1,406 Days
• Field of Search: 704216-218, 704/237, 704/257, 704/263, 704/9, 704/214, 704/235, 704/270.1, 704/275
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces

Abstract

A device, method and system are provided for interpreting and executing operations based on multimodal input received at a computing device. The multimodal input can include one or more verbal and non-verbal inputs, such as a combination of speech and gesture inputs received substantially concurrently via suitable user interface means provided on the computing device. One or more target objects is identified from the non-verbal input, and text is recognized from the verbal input. An interaction object is generated using the recognized text and identified target objects, and thus comprises a natural language expression with embedded target objects. The interaction object is then processed to identify one or more operations to be executed.

Claims

1. A method implemented at a computing device, the method comprising:

receiving verbal input using a verbal input interface of the computing device;

receiving, concurrently with at least part of the verbal input, at least one secondary input using a non-verbal input interface of the computing device, the non-verbal input interface being selected from the group of;

a kinetic input interface, an inertial input interface, a perceptual input interface, a touch input interface, a graphical user interface, and a sensor input interface;

identifying one or more target objects from the at least one secondary input;

recognizing text from the received verbal input;

generating an interaction object, the interaction object comprising a natural language expression having references to the one or more identified target objects identified from the at least one secondary input, the references being embedded within the recognized text, the generating of the interaction object comprising identifying at least one attribute associated with each of the one or more identified target objects or at least one operation associated with each of the one or more identified target objects;

processing the interaction object to identify at least one operation to be executed on at least one of the one or more identified target objects; and

executing the operation on the at least one of the one or more identified target objects.

2. The method of claim 1, wherein the one or more target objects are identified prior to completion of recognition of the text from the received verbal input.

3. The method of claim 1, wherein each of the one or more identified target objects is associated with a metaobject defining the associated at least one attribute or at least one operation.

4. The method of claim 1, wherein processing the interaction object comprises correlating at least a part of the recognized text with at least one identified attribute of at least one of the one or more identified target objects.

5. The method of claim 1, further comprising:

displaying a text or graphical representation of the interaction object for user confirmation prior to processing the interaction object;

receiving an indication of an error in the text recognized from the received verbal input; and

providing a selection of one or more options to correct the indicated error, the one or more options being determined from at least one attribute associated with the one or more identified target objects.

6. The method of claim 1, further comprising sending the interaction object to a further computing device for processing.

7. The method of claim 1, wherein the at least one secondary input comprises a touch-based input.

8. The method of claim 1, wherein the verbal input comprises speech input.

9. The method of claim 1, wherein the verbal input comprises text input.

10. The method of claim 1, wherein the interaction object comprises a plurality of operations to be executed on the at least one of the one or more identified target objects, the method further comprising:

executing a first one of the plurality of operations on the at least one of the one or more identified target objects while buffering remaining ones of the plurality of operations; and

sequentially executing the remaining ones of the plurality of operations after execution of the first one of the plurality of operations.

11. The method of claim 1, further comprising:

storing the interaction object; and

subsequently, at the computing device or another computing device, processing the interaction object to identify the at least one operation to be executed on at least one of the one or more identified target objects, and executing the operation on the at least one of the one or more identified target objects.

12. The method of claim 1, wherein the computing device executes a plurality of applications, and the interaction object references one or more identified target objects from the plurality of applications.

13. The method of claim 12, wherein the plurality of applications are executing in a split screen mode on the computing device.

14. A computing device, comprising:

at least one verbal input interface;

at least one non-verbal input interface being selected from the group of;

a kinetic input interface, an inertial input interface, a perceptual input interface, a touch input interface, a graphical user interface, and a sensor input interface;

at least one processor in communication with the at least one verbal input interface and the at least one non-verbal input interface, the at least one processor being configured to;

receive verbal input using the verbal input interface;

receive, concurrently with at least part of the verbal input, at least one secondary input using the at least one non-verbal input interface;

identify one or more target objects from the at least one secondary input;

recognize text from the received verbal input;

generate an interaction object, the interaction object comprising a natural language expression having references to the one or more identified target objects identified from the at least one secondary input, the references being embedded within the recognized text, the generation of the interaction object comprising identification of at least one attribute associated with each of the one or more identified target objects or at least one operation associated with each of the one or more identified target objects;

process the interaction object to identify at least one operation to be executed on at least one of the one or more identified target objects; and

execute the operation on the at least one of the one or more identified target objects.

15. The computing device of claim 14, wherein the one or more target objects are identified prior to completion of recognition of the text from the received verbal input.

16. The computing device of claim 14, wherein each of the one or more identified target objects is associated with a metaobject defining the associated at least one attribute or at least one operation.

17. The computing device of claim 14, wherein the at least one processor is configured to process the interaction object by correlating at least a part of the recognized text with at least one identified attribute of at least one of the one or more identified target objects.

18. The computing device of claim 14, wherein the at least one processor is further configured to:

display a text or graphical representation of the interaction object for user confirmation on a display of the computing device, prior to processing the interaction object;

receive an indication of an error in the text recognized from the received speech input; and

provide a selection of one or more options to correct the indicated error, the one or more options being determined from at least one attribute associated with the one or more identified target objects.

19. The computing device of claim 14, wherein the at least one processor is further configured to send the interaction object to a further computing device for processing.

20. The computing device of claim 14, wherein the at least one secondary input comprises a touch-based input.

21. The computing device of claim 14, wherein the verbal input comprises speech input.

22. The computing device of claim 14, wherein the verbal input comprises text input.

23. The computing device of claim 14, wherein the interaction object comprises a plurality of operations to be executed on the at least one of the one or more identified target objects, and the at least one processor is further configured to:

execute a first one of the plurality of operations on the at least one of the one or more identified target objects while buffering remaining ones of the plurality of operations; and

sequentially execute the remaining ones of the plurality of operations after execution of the first one of the plurality of operations.

24. The computing device of claim 14, wherein the at least one processor is further configured to:

store the interaction object; and

subsequently process the interaction object to identify the at least one operation to be executed on at least one of the one or more identified target objects, and execute the operation on the at least one of the one or more identified target objects.

25. The computing device of claim 14, wherein the at least one processor is further configured to execute a plurality of applications, and the interaction object references one or more identified target objects from the plurality of applications.

26. The computing device of claim 25, wherein the plurality of applications are executing in a split screen mode on the computing device.

27. A non-transitory computer-readable medium bearing code which, when executed by one or more processors of a computing device, causes the computing device to implement a method of:

receiving verbal input using a verbal input interface of the computing device;

receiving, concurrently with at least part of the verbal input, at least one secondary input using a non-verbal input interface of the computing device, the non-verbal input interface being selected from the group of;

a kinetic input interface, an inertial input interface, a perceptual input interface, a touch input interface, a graphical user interface, and a sensor input interface;

identifying one or more target objects from the at least one secondary input;

recognizing text from the received verbal input;

generating an interaction object, the interaction object comprising a natural language expression having references to the one or more identified target objects identified from the at least one secondary input, the references being embedded within the recognized text, the generating of the interaction object comprising identifying at least one attribute associated with each of the one or more identified target objects or at least one operation associated with each of the one or more identified target objects;

processing the interaction object to identify at least one operation to be executed on at least one of the one or more identified target objects; and

executing the operation on the at least one of the one or more identified target objects.