Controlled computer interface

• US 5,377,303 A
• Filed: 12/09/1993
• Issued: 12/27/1994
• Est. Priority Date: 06/23/1989
• Status: Expired due to Term

• Current Assignee: Multimodal Technologies Incorporated (3M Company)
• Original Assignee: Articulate Systems, Inc. (Lernout & Hauspie NV Speechproducts)
• Inventors: Firman, Thomas R.
• Primary Examiner(s): KNEPPER, DAVID D
• Assistant Examiner(s): KNEPPER, DAVID D
• Application Number:
  US08/165,014
• Time in Patent Office: 383 Days
• Field of Search: 381/41-45, 395/2.6, 395/2.84
• US Class Current: 704/275
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Voice utterances are substituted for manipulation of a pointing device, the pointing device being of the kind which is manipulated to control motion of a cursor on a computer display and to indicate desired actions associated with the position of the cursor on the display, the cursor being moved and the desired actions being aided by an operating system in, the computer in response to control signals received from the pointing device, the computer also having an alphanumeric keyboard, the operating system being separately responsive to control signals received from the keyboard in accordance with a predetermined format specific to the keyboard; in the system, a voice recognizer recognizes the voiced utterance, and an interpreter converts the voiced utterance into control signals which will directly create a desired action aided by the operating system without first being converted into control signals expressed in the predetermined format specific to the keyboard. In another aspect, voiced utterances are converted to commands, expressed in a predefined command language, to be used by an operating system of a computer, by converting some voiced utterances into commands corresponding to actions to be taken by the operating system, and converting other voiced utterances into commands which carry associated text strings to be used as part of text being processed in an application program running under the operating system.

Claims

1. A system for enabling voiced utterances to control window elements in a graphical user interface, said graphical user interface being provided by an operating system responsive to events posted in an event queue, some events in the queue being posted in response to signals received from an alphanumeric keyboard in accordance with a predetermined format specific to the keyboard, said events including higher level events, comprising

a voice recognizer for recognizing voiced utterances, and

an interpreter functionally connected to said voice recognizer for

converting at least some of the voiced utterances into said higher level events for controlling said window elements and

posting said higher level events to the event queue, without first converting said voiced utterances into signals expressed in the predetermined format specific to the keyboard.

2. The system of claim 1 wherein said higher level events posted by said interpreter mimic events fed to said event queue by a mouse.

3. The system of claim 1 wherein said one of said higher level events directs said program to wait for a predetermined time delay.

4. The system of claim 1 wherein said interpreter converts at least some of said voiced utterances to said higher level events based on each of said voiced utterances and on a state of said program.

5. The system of claim 4 wherein said interpreter further comprises

stored data controlling said conversion of said voiced utterance to said higher level event, and

means for generating a portion of said stored data by examining said program.

6. The system of claim 5 wherein said data are generated by examining menus and control buttons of an executable image of said program.

7. The system of claim 4 wherein said interpreter further comprises stored data controlling said conversion of said voiced utterances to said higher level events,

means for viewing and editing said stored data.

8. The system of claim 1 further comprising

stored data controlling said conversion of said voiced utterances to said higher level events, and

an event recorder for generating a portion of said data by said event recorder examining an execution session of said program.

9. The system of claim 8 wherein said event recorder is implemented by code substituted for the code normally executed by a trap handler of said operating system.

10. The system of claim 9 wherein said event recorder examines the state of data structures maintained by said operating system.

11. The system of claim 8, wherein said event recorder can be rerun to incrementally re-generate a portion of said data.

12. The system of claim 8 further comprising

a pointing device to control a location indicator on a display,

means to control said event recorder with said pointing device, and

means within said event recorder to distinguish pointer movements and pointer device button presses as either intended to produce commands to said program or to control said event recorder.

13. The system of claim 12 wherein said distinguishing means comprises a global variable tracking the state of the buttons of said pointer device.

14. The system of claim 1 further adapted to enable voiced utterances to be substituted for manipulation of a pointing device to control motion of a displayed location indicator on a computer display, the indicator being moved by an operating system in a computer in response to control signals received from the pointing device, and wherein said interpreter is further connected to said voice recognizer for converting voiced utterances into events which will cause desired movements of the indicator aided by the operating system.

15. The system of claim 14 further comprising a program for execution with said operating system, a state of said program comprising a configuration on said display, and wherein said higher level events posted by said interpreter direct motion of said indicator relative to said configuration.

16. The system of claim 15 wherein said configuration on said display comprises characters.

17. The system of claim 15 wherein said higher level events posted by said interpreter further direct said location indicator to the screen position said location indicator indicated immediately before said voiced utterance was recognized.

18. The system of claim 14 wherein one of said higher level events directs said location indicator to indicate a position specified by a local window-relative coordinate.

19. The system of claim 14 wherein one of said higher level events directs the location indicator to indicate a position specified by a global screen-absolute coordinate.

20. The system of claim 14 wherein one of said higher level events directs the location indicator to indicate a specified screen button or dialog box.

21. The system of claim 14 wherein one of said high level events directs the indicator to move from a current position (y,x) to a new position (y+δ

y,x+δ

x).

22. The system of claim 14 wherein one of said high level events directs the location indicator to move continuously by a (δ

y, δ

x) predetermined incremental distance per predetermined time interval.

23. The system of claim 22 wherein said one high level event is generated during a timer interrupt of said operating system, said timer interrupt occurring on the order of ten to one hundred times per second.

24. The system of claim 14 wherein said program provides user menu selections to be selected by pointer device movements and/or button presses, and wherein said interpreter produces a series of higher level events in response to said pointer device movements and/or button presses.

25. The system of claim 14 or 1 wherein said operating system is an operating system of a Macintosh computer, and said event queue is an event queue of said Macintosh operating system.

26. The system of claim 14 or 1 wherein said window elements comprise zooming windows.

27. The system of claim 14 or 1 wherein said window elements comprise moving windows nearer to or farther from the front of a set of windows.

28. The system of claim 14 or 1 wherein said voiced utterances are converted into events which will cause movement of the indicator in a desired direction aided by the operating system in the computer, said movement continuing unabated until stopped by an action of the user.