METHODS AND SYSTEMS FOR MULTIMODAL INTERACTION

• US 20150363047A1
• Filed: 02/07/2014
• Published: 12/17/2015
• Est. Priority Date: 02/14/2013
• Status: Abandoned Application

• Current Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Original Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Inventors: Mathur, Akhil
• Application Number:
  US14/767,715
• Publication Number:
  US 20150363047A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

Methods and systems for multimodal interaction are described herein. In one embodiment, a method for multimodal interaction comprises determining whether a first input modality is successful in providing inputs for performing a task. The method further includes prompting the user to use a second input modality to provide inputs for performing the task on determining the first input modality to be unsuccessful. Further, the method comprises receiving inputs from at least one of the first input modality and the second input modality. The method further comprises performing the task based on the inputs received from at least one of the first input modality and the second input modality.

Claims

1. A method for multimodal interaction comprising:

determining whether a first input modality is successful in providing inputs for performing a task;

prompting the user to use a second input modality to provide the inputs for performing the task when the first input modality is unsuccessful;

receiving the inputs from at least one of the first input modality and the second input modality; and

performing the task based on the inputs received from at least one of the first input modality and the second input modality.

2. The method as claimed in claim 1, wherein the determining comprises:

receiving, through the first input modality, the input from the user for performing the task;

determining whether the input is executable for performing the task;

increasing a value of an error count by one for the input being non-executable for performing the task, wherein the error count is a count of a number of inputs received from the first input modality for performing the task;

comparing the error count with a threshold value; and

determining the first input modality to be unsuccessful for the error count being greater than the threshold value.

3. The method as claimed in claim 1, wherein the determining comprises:

receiving, through the first input modality, the input from a user for performing the task;

ascertaining whether the input is executable for performing the task; and

determining the first input modality to be successful for the input being executable for performing the task.

4. The method as claimed in claim 1 further comprises selecting an input modality from among a plurality of input modalities as the second input modality based on predefined rules.

5. The method as claimed in claim 4, wherein the predefined rules include at least one of a predetermined order of using input modalities, random selection of the second input modality from among the plurality of input modalities, and ascertaining the second input modality based on the type of the first input modality.

6. The method as claimed in claim 1, wherein the prompting the user to use the second input modality further comprises providing a list of input modalities to allow the user to select the second input modality.

16. A computer program product comprising a computer readable medium, having thereon a computer program comprising program instructions, the computer program being loadable into a data-processing unit and adapted to cause execution of the method according to claim 1 when the computer program is run by the data-processing unit.

17. A computer program adapted to perform the methods in accordance with claim 1.

7. A multimodal interaction system configured to:

determine whether a first input modality is successful in providing inputs for performing a task;

prompt the user to use a second input modality to provide the inputs for performing the task when the first input modality is unsuccessful;

receive the inputs from at least one of the first input modality and the second input modality; and

perform the task based on the inputs received from at least one of the first input modality and the second input modality.

8. The multimodal interaction system as claimed in claim 7, wherein the multimodal interaction system is further configured to:

receive, through the first input modality, the input from the user for performing the task;

determine whether the input is executable for performing the task;

increase a value of an error count by one for the input being non-executable for performing the task, wherein the error count is a count of a number of inputs received from the first input modality for performing the task;

compare the error count with a threshold value; and

determine the first input modality to be unsuccessful for the error count being greater than the threshold value.

9. The multimodal interaction system as claimed in claim 7, wherein the multimodal interaction system is further configured to:

receive, through the first input modality, the input from a user for performing the task;

ascertain whether the input is executable for performing the task; and

determine the first input modality to be successful for the input being executable for performing the task.

10. The multimodal interaction system as claimed in claim 7, wherein the multimodal interaction system is further configured to select an input modality from among a plurality of input modalities as the second input modality based on predefined rules.

11. The multimodal interaction system as claimed in claim 10, wherein the predefined rules include at least one of a predetermined order of using input modalities, random selection of the second input modality from among the plurality of input modalities, and ascertaining the second input modality based on the type of the first input modality.

12. The multimodal interaction system as claimed in claim 7, wherein the multimodal interaction system is further configured to provide a list of input modalities to allow the user to select the second input modality.

13. The multimodal interaction system as claimed in claim 7, wherein the multimodal interaction system is further configured to display at least one of a name of the second input modality and an icon indicating the second input modality to prompt the user to use the second input modality.

14. The multimodal interaction system as claimed in claim 7, wherein the multimodal interaction system comprises:

a processor;

an interaction module coupled to the processor, the interaction module configured to;

receive the inputs from at least one of the first input modality and the second input modality;

an inference module coupled to the processor, the inference module configured to;

determine whether a first input modality is successful in providing inputs for performing a task;

prompt the user to use a second input modality to provide the inputs for performing the task when the first input modality is unsuccessful; and

perform the task based on the inputs received from at least one of the first input modality and the second input modality.

15. A computing system comprising the multimodal interaction system as claimed in claim 7, wherein the computing system is one of a desktop computer, a hand-held device, a multiprocessor system, a personal digital assistant, a mobile phone, a laptop, a network computer, a cloud server, a minicomputer, a mainframe computer, a touch-enabled camera, and an interactive gaming console.