Apparatus for managing a multi-modal user interface

• US 20020178344A1
• Filed: 05/22/2002
• Published: 11/28/2002
• Est. Priority Date: 05/22/2001
• Status: Abandoned Application

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Bourguet, Marie-Luce, Jost, Uwe Helmut
• Application Number:
  US10/152,284
• Publication Number:
  US 20020178344A1
• Time in Patent Office: Days
• US Class Current: 712/1
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The apparatus has a receiver (200) for receiving input events from at least two different modality modules (23 to 30); a plurality of instruction determining units (201b) each arranged to respond to a specific input event or specific combination of input events; and a supplier (201a) supplying events received by the receiver to the instruction determining units (23 to 30), wherein each instruction determining unit (23 to 30) is operable to supply a signal for causing a corresponding instruction to be issued when the specific input event or specific combination of input events to which that instruction determining unit is responsive is received by that instruction determining unit.

Claims

1. Apparatus for managing a multi-modal interface, which apparatus comprises:

receiving means for receiving events from at least two different modality modules;

a plurality of instruction determining means each arranged to respond to a specific event or specific combination of events; and

supplying means for supplying events received by the receiving means to the instruction determining means, wherein each instruction determining means is operable to supply a signal for causing a corresponding instruction to be issued when the specific event or specific combination of events to which that instruction determining means is responsive is received by that instruction determining means.

2. Apparatus according to claim 1, wherein the supplying means comprises event type determining means for determining the modality of a received event and for supplying the received event to the or each instruction determining means that is responsive to an event of that modality or to a combination of events including an event of that modality.

3. Apparatus according to claim 1, wherein when an instruction determining means is arranged to be responsive to a specific combination of events, the instruction determining means is arranged to be responsive to that specific combination of events if the events of that combination are all received within a predetermined time.

4. Apparatus according to claim 1, wherein each instruction determining means is arranged to switch itself off if a received event is one to which it is not responsive until another instruction determining means has supplied a signal for causing an instruction to be issued.

5. Apparatus according to claim 1, further comprising priority determining means for determining a signal priority when two or more of the instruction determining means supply signals at the same time.

6. Apparatus according to claim 5, wherein the priority determining means is arranged to determine a signal priority using confidence data associated with the signal.

7. Apparatus according to claim 1, further comprising command generation means for receiving signals from said instruction determining means and for generating a command corresponding to a received signal.

8. Apparatus according to any one of the preceding claims, further comprising event managing means for listening for events and for supplying events to the receiving means.

9. Apparatus according to claim 1, further comprising at least one operation means controllable by instructions caused to be issued by a signal from an instruction determining means.

10. Apparatus according to claim 9, wherein said at least one operation means comprises means running application software.

11. Apparatus according to claim 9, wherein said at least one operation means comprises control circuitry for carriing out a function.

12. Apparatus according to claim 11, wherein the control circuitry comprises control circuitry for carrying out a photocopying function.

13. Apparatus according to claim 9, wherein said at least one operation means comprises dialog means for conducting a multi-modal dialog with a user, wherein a dialog state of said dialog means is controllable by instructions caused to be issued by said instruction determining means.

14. Apparatus according to claim 1, further comprising managing means responsive to instructions received from an application or dialog to determine the event or combination of events to which the instruction determining means are responsive.

15. Apparatus according to claim 1, further comprising control means for modifying an event or changing a response to an event from one modality module in accordance with an event from another modality module or modules.

16. Apparatus according to claim 1, further comprising means for providing a signal to one modality module to cause that modality module to modify its processing in dependence upon an event received from another modality module or modules.

38. A multi-modal interface having apparatus in accordance with claim 1.

40. A processor-controlled machine having apparatus in accordance with claim 1.

42. A signal carrying processor instructions for causing a processor to implement a method in accordance with claim 1.

43. A storage medium carrying processor implementable instructions for causing processing means to implement a method in accordance with claim 1.

17. Apparatus for managing a multi-modal interface, which apparatus comprises:

a plurality of instruction providing means for each providing a specific different instruction for causing an application to carry out a specific function, wherein each instruction providing means is arranged to respond only to a specific combination of multi-modal events so that an instruction providing means is arranged to issue its instruction only when that particular combination of multi-modal events has been received.

18. Apparatus for managing a multi-modal interface, which apparatus comprises:

means for receiving events from at least two different modality modules; and

processing means for processing events received from the at least two different modality modules, wherein the processing means is arranged to modify an event or change its response to an event from one modality module in dependence upon an event from another modality module or modality modules.

19. Apparatus for managing a multi-modal interface, which apparatus comprises:

means for receiving events from at least two different modality modules; and

processing means for processing events received from the at least two different modality modules, wherein the processing means is arranged to process an event from one modality module in accordance with an event from another modality module or modules and to provide a feedback signal to the one modality module to cause it to modify its processing of a user input in dependence upon an event from another modality module or modules.

20. A method of operating a processor apparatus to manage a multi-modal interface, which method comprises the processor apparatus carrying out the steps of:

receiving events from at least two different modality modules;

providing a plurality of instruction determining means each arranged to respond to a specific event or specific combination of events; and

supplying received events to the instruction determining means so that an instruction determining means supplies a signal for causing a corresponding instruction to be issued when the specific event or specific combination of events to which that instruction determining means is responsive is received.

21. A method according to claim 20, wherein the supplying step comprises determining the modality of a received event and supplying the received event to the or each instruction determining means that is responsive to an event of that modality or to a combination of events including an event of that modality.

22. A method according to claim 20, wherein when an instruction determining means is responsive to a specific combination of events, the instruction determining means responds to that specific combination of events if the events of that combination are all received within a predetermined time.

23. A method according to claim 20, wherein each instruction determining means switches itself off if a received event is one to which it is not responsive until another instruction determining means has supplied a signal for causing an instruction to be issued.

24. A method according to claim 20, further comprising determining a signal priority when two or more of the instruction determining means supply signals at the same time.

25. A method according to claim 20, further comprising receiving signals from said instruction determining means and generating a command corresponding to a received signal.

26. A method according to claim 20, wherein the receiving step comprises listening for events and supplying events to the receiving means.

27. A method according to claim 20, further comprising controlling at least one operation means by instructions caused to be issued by a signal from an instruction determining means.

28. A method according to claim 27, wherein the controlling step comprises controlling at least one operation means comprising means running application software.

29. A method according to claim 27, wherein the controlling step controls at least one operation means comprising control circuitry for carrying out a function.

30. A method according to claim 29, wherein the controlling step comprises controlling control circuitry for carrying out a photocopying function.

31. A method according to claim 27, wherein said controlling step controls at least one operation means comprising dialog means for conducting a multi-modal dialog with a user so that a dialog state of said dialog means is controlled by instructions caused to be issued by said instruction determining means.

32. A method according to claim 20, further comprising the step of determining the event or combination of events to which the instruction determining means are responsive in accordance with instructions received from an application or dialog.

33. A method according to claim 20, further comprising the step of modifying an event or changing a response to an event from one modality module in accordance with an event from another modality module or modules.

34. A method according to claim 20, further comprising the step of providing a signal to one modality module to cause that modality module to modify its operation in dependence upon an event received from another modality module or modules.

39. A processor-controlled machine having a multi-modal interface in accordance with claim 38.

41. A processor-controlled machine according to claim 39 arranged to carry out at least one of photocopying and facsimile functions.

35. A method of operating a processor apparatus to manage a multi-modal interface, which method comprises the processor apparatus providing a plurality of instruction providing means for each providing a specific different instruction for causing an application to carry out a specific function, so that each instruction providing means responds only to a specific combination of multi-modal events and issues its instruction only when that particular combination of multi-modal events has been received.

36. A method of operating a processor apparatus to manage a multi-modal interface, which method comprises the processor apparatus carrying out the steps of:

receiving events from at least two different modality modules;

processing events received from the at least two different modality modules, and modifying an event or changing its response to an event from one modality module in dependence upon an event from another modality module or modality modules.

37. A method of operating a processor apparatus to manage a multi-modal interface, which method comprises the processor apparatus carrying out the steps of:

receiving events from at least two different modality modules; and

providing a feedback signal to the one modality module to cause it to modify its processing in dependence upon an event from another modality module or modules.

44. Apparatus for managing a multi-modal interface, which apparatus comprises:

a receiver for receiving events from at least two different modality modules;

a plurality of instruction determining units each arranged to respond to a specific event or specific combination of events; and

a supplier for supplying events received by the receiver to the instruction determining units, wherein each instruction determining unit is operable to supply a signal for causing a corresponding instruction to be issued when the specific event or specific combination of events to which that instruction determining unit is responsive is received by that instruction determining unit.

45. Apparatus for managing a multi-modal interface, which apparatus comprises:

a plurality of instruction providing units for each providing a specific different instruction for causing an application to carry out a specific function, wherein each instruction providing unit is arranged to respond only to a specific combination of multi-modal events so that an instruction providing unit is arranged to issue its instruction only when that particular combination of multi-modal events has been received.

46. Apparatus for managing a multi-modal interface, which apparatus comprises:

a receiver for receiving events from at least two different modality modules; and

a processor for processing events received from the at least two different modality modules, wherein the processor is arranged to modify an event or change its response to an event from one modality module in dependence upon an event from another modality module or modality modules.

47. Apparatus for managing a multi-modal interface, which apparatus comprises:

a receiver for receiving events from at least two different modality modules; and

a processor for processing events received from the at least two different modality modules, wherein the processor is arranged to process an event from one modality module in accordance with an event from another modality module or modules and to provide a feedback signal to the one modality module to cause it to modify its processing in dependence upon an event from another modality module or modules.