Contextual speech-recognition user-interface driven system and method

• US 9,471,333 B2
• Filed: 11/02/2007
• Issued: 10/18/2016
• Est. Priority Date: 11/03/2006
• Status: Expired due to Fees

• Current Assignee: Philippe Roy
• Original Assignee: Conceptual Speech LLC
• Inventors: Roy, Philippe
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): HEFFINGTON, JOHN M
• Application Number:
  US11/934,661
• Publication Number:
  US 20100138759A1
• Time in Patent Office: 3,273 Days
• Field of Search: 715/727, 715/788, 715/817, 715/825, 715/803
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

In an effort to customize or enhance software applications, configuration data is often used. Configuration settings that are editable by users need not to be limited to a simple flat entry that can be taken out of context anymore. The present invention allows for multiple-levels of configuration settings to interact with each other, so that a single configuration for a given context to be calculated dynamically. In the process, the user gains flexibility to specify more adequately a required change or customization while propagating the information with minimal effort and not requiring additional coding. Furthermore, to simplify a speaker'"'"'s interactions for controlling an automated device, the addition of a superposed layer over graphic user interface may be used. The superposed layer may display coordinates that a speaker may use to navigate the graphic user interface, for example to associate a location with a keyword or a coordinate.

Claims

1. A method for interaction with a computer software application having a native interface having a set of dynamically changing user interface elements for interaction with a human user, executing under control of an operating system on an automated processing system, having a context comprising a focus, the operating system selectively providing received user inputs from a human user to a respective user interface element having the focus, comprising:

determining the context of the operating system executing the at least one software application, based on at least an analysis of the set of dynamically changing user interface elements of the native interface of the computer software application;

defining, through the operating system, a supplemental interface for interaction between the human user and an automated user interface system with the at least one computer software application, selected from a plurality of available supplemental interfaces, each respective supplemental interface supporting at least one human user input unsupported by the respective changing user interface element available in the native interface in the context, a respective supplemental interface being selectively made available through the automated user interface system for the respective user interface element having the focus in dependence on the determined context comprising the focus; and

communicating with the computer software application having focus using at least the input unsupported by the native interface of the respective defined supplemental interface,

wherein the computer software application receives input from the respective defined supplemental interface using at least the input unsupported by the native interface without requiring either access to a source code of the computer software application, or modification to the computer software application,

wherein the at least one input unsupported by the native interface comprises a speech input corresponding to a graphic user interface command of the native interface.

2. The method according to claim 1, wherein the determined context has a hierarchy, and wherein the respective supplemental interface is defined based on a plurality of sets of configurations having a priority varying according to the hierarchy.

3. The method according to claim 1, wherein the determined context comprises at least one set of dynamically changing available commands of the native interface.

4. The method according to claim 1, wherein the determined context comprises a set of open windows within a graphic user interface of the native interface.

5. The method according to claim 1, wherein the determined context is associated with a set of configurations, each respective configuration comprising a predetermined file associated with a respective dynamically presented object of the native interface.

6. The method according to claim 1, wherein the determined context is associated with at least one set of configurations comprising speech commands for control of the computer software application.

7. The method according to claim 1, wherein said communicating comprises presenting a message comprising a text string to the computer software application.

18. The method according to claim 1, wherein at least one of the set of dynamically changing user interface elements is repositionable, further comprising selectively repositioning the repositionable at least one of the set of dynamically changing user interface elements in response to a speech command defining the repositioning, received through the supplemental interface.

8. A system providing a modified interaction of a human user with a computer software application having a native interface, executing under an operating system having a determinable context comprising at least a dynamically changing state of a human user interface comprising a focus representing a user interface element which receives human user input, the system comprising:

a plurality of supplemental interfaces for interaction by the human user with the at least one computer software application, each supplemental interface supporting at least one human user input type which is unsupported by a respective associated human user interface element of the native interface; and

a processor configured to select at least one of the plurality of supplemental interfaces based on at least the context which is appropriate for the human user interface element which has the focus, and to communicate in relation to the computer software application through the at least one of the plurality of supplemental interfaces using at least the at least one input type which is supported by the selected supplemental interface and unsupported by the native interface, wherein a human user interaction with the native interface is modified by use of the at least one of the plurality of supplemental interfaces to permit use of the at least one human user input type which is unsupported by the native interface without requiring either access to the source code of the computer software application, or modification to the computer software application,

wherein the native interface specifies receipt of human user input through a graphic user interface as a pointing device input or a keyboard input, and at least one of the plurality of supplemental interfaces is configured to receive a speech input and to present information selectively corresponding to the speech input to the native input as a command.

9. The system according to claim 8, wherein the context has a hierarchy, and wherein the supplemental interface is defined based on a plurality of sets of configurations having a priority varying according to the hierarchy.

10. The system according claim 8, wherein the context comprises at least one set of dynamically changing available commands of the native interface.

11. The system according to claim 8, wherein the context comprises a set of open windows within a graphic user interface of the native interface, wherein the focus determines which of the set of open windows receives the user input of the at least one human user input type which is supported by the selected supplemental interface and unsupported by the native interface.

12. The system according to claim 8, wherein a respective supplemental interface comprises a set of configurations, each configuration comprising a predetermined file defining a set of user-selectable options associated with a dynamically selectively presented user interface object in the native interface.

13. The system according to claim 8, wherein a respective supplemental interface comprises a set of configurations, each respective configuration comprising a data file representing parameters for processing speech input of the supplemental interface corresponding to a graphic user interface command of the native interface.

14. The system according to claim 8, wherein the at least one the plurality of supplemental interfaces comprises a set of speech commands.

15. The system according to claim 8, wherein the processor is configured to communicate by at least presenting a message comprising a text string generated by a respective supplemental interface to the native interface of the computer software application.

16. A graphic user interface computer system, communicating with a display device configured to present to a human user a set of graphic objects defined by at least one software application executing under control of an operating system having a context comprising a focus which dynamically associates a human user input with a respective graphic object of the set of graphic objects, having a native graphic user interface, comprising:

a speech recognition engine executing under the operating system, configured to receive user speech commands and to output at least one control signal for the at least one software application;

a voice controlled graphic layer executing under the operating system, superposed on, or blended over, the native graphic user interface or a representation of the graphic user interface of the at least one software application, the voice controlled graphic layer supporting at least one human user input unsupported by the native graphic user interface, the voice controlled graphic layer being defined independently of the source code of the at least one software application, configured to present a graphic indication to the human user of available speech commands for the respective graphic object of the set of graphic objects of the at least one software application selectively in response to a determination of which graphic object has focus, wherein the superposed voice controlled graphic layer does not disrupt a clarity of the native graphic user interface; and

an input port configured to receive a speech input to the speech recognition engine.

17. A method of speech enabling a software application defining a native graphic user interface having a plurality of distinct graphic user interface events, for execution under a computer operating system, within an automated processing system, having a context defined by at least a graphic user interface status of the computer operating system and any applications executing under the computer operating system, the operating system having a focus for controlling a respective graphic human user interface element which is associated with a human user input, comprising:

generating a native graphic user interface of the software application;

overlaying, by a supplemental application distinct from the software application, executing under the computer operating system, a plurality of identified regions on the native graphic user interface in dependence on the context and independent of a source code of the software application;

receiving a speech input from a human user, identifying a selected identified region, corresponding to a graphic human user interface element of the native graphic user interface of the software application, to grant focus to the corresponding graphic human user interface element if it does not have focus; and

generating, selectively in dependence on an analyzed content of the speech input, at least one graphic user interface event supported by the native graphic user interface associated with the graphic human user interface element corresponding to the selected identified region.

19. The method according to claim 17, wherein at least one of the plurality of distinct graphic user interface events is responsive to pointing device commands received through the native graphic user interface, further comprising emulating the pointing device commands dependent on the received speech input by the supplemental application.

20. The method according to claim 17, wherein the defined context is associated with a configuration file comprising predetermined parameters associated with a state of the respective graphic human user interface element having the focus.