MULTIMODAL COMPUTER NAVIGATION

• US 20090049388A1
• Filed: 06/02/2006
• Published: 02/19/2009
• Est. Priority Date: 06/02/2005
• Status: Abandoned Application

• Current Assignee: National ICT Australia Limited
• Original Assignee: National ICT Australia Limited
• Inventors: Shi, Yu, Taib, Ronnie Bernard Francis, Chen, Fang
• Application Number:
  US11/916,255
• Publication Number:
  US 20090049388A1
• Time in Patent Office: Days
• US Class Current: 715/738
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

This invention concerns multimodal computer navigation, that is operation of a computer using traditional modes such as keyboard together with less conventional modes such as speech and gestures. The invention has particular application for navigation of information presentations, such as webpages and database user interfaces, and is presented as a method, a browser, software and a computer system. The information navigated is not described in a multimodal way. Two or more unimodal navigation signals are received from a user and interpreted. These interpretations are fused to automatically determining the user'"'"'s intended navigation selection.

Claims

1. A method for multimodal computer navigation, suitable for navigating information presentations where the information navigated is not described in a multimodal way;

the method comprising the steps of;

receiving unimodal navigation signals from a user;

receiving other unimodal navigation signals from the user;

interpreting the navigation signals;

interpreting the other navigation signals; and

automatically determining the user'"'"'s intended navigation selection from a fusion of both interpretations.

2. A method according to claim 1, wherein one of the unimodal navigation signals is generated from a conventional input device.

3. A method according to claim 2, wherein the other unimodal navigation signals is generated from speech or a body gesture.

4. A method according to claim 3, wherein the body gestures include movements of the head, hand and other body parts such as eyes.

5. A method according to claim 3, wherein the body gestures are captured by analysing video, or from motion transducers worn by the user.

6. A method according to claim 1, the method further comprising the step of predefining fusions of unimodal signals that form a navigation selection.

7. A method according to claim 6, wherein personal or task oriented profiles are created for particular users or tasks.

8. A method according to claim 1, the method further comprising determining the possible navigation selections that could be selected by the user for the information presentation.

9. A method according to claim 8, wherein the step of determining the possible navigation selections is repeated for every information presentation that is displayed to the user.

10. A method according to claim 1, wherein the information presentation is a graphical display of information and the user'"'"'s selected navigation is either navigation of the entire display or of a smaller information presentation within the information presentation.

11. A method according to claim 1, comprising the further step of learning and adapting to a particular user.

12. A method according to claim 1, wherein fusion involves generating some combination of the interpretations, and using a resulting combination signal to make the automatic determination.

13. A method according to claim 1, wherein fusion involves sequential consideration of interpretations of transducer generated and body gesture navigation signals.

14. A method according to claim 13, comprising the further step of responding to an earlier inconclusive interpretation in some way before receiving or taking account of a later inconclusive interpretation.

15. A method according to claim 14, wherein the responding step involves changing the display and then receiving further unimodal navigation signals from a user to form a conclusive interpretation.

28. A computer browser programmed to perform the method of claim 1.

29. A software program to perform the method of claim 1.

30. A software program according to claim 29, wherein the software program is incorporated with the operating system software of a computer system.

31. A software program according to claim 29, wherein the software program is incorporated with application software.

32. A computer system programmed to perform the method of claim 1.

33. A method according to claim 1, wherein one of the unimodal navigation signals is generated from body gestures, and the other unimodal signals is generated from speech of the user.

34. A method according to claim 33, wherein the step of automatically determining the user'"'"'s intended navigation selection from a fusion of both interpretations comprises identifying the navigation selections within the information presentation and on a determined extended trajectory in the direction of the body gesture and selecting the navigation selection that passes through the trajectory that is described by the speech of the user.

35. A method according to claim 34, wherein the step of determining the trajectory further comprises moving a pointer within the information presentation along the trajectory.

36. A method according to claim 34, wherein the method further comprises the initial step of determining all the possible navigation selections that could be selected by the user for the information presentation.

16. A computer system suitable for use with multimodal navigation of information presentations where the information navigated is not described in a multimodal way;

the computer system comprising;

display means to display information presentations to a user;

input means to receive two or more unimodal navigation signals from the user; and

processing means to interpret the two or more unimodal navigations signals and to automatically determine the user'"'"'s intended navigation selection from a fusion of both interpretations.

17-27. -27. (canceled)