Contextual speech-recognition user-interface driven system and method
First Claim
1. A method for interaction with a computer software application having a native interface having a set of dynamically changing user interface elements for interaction with a human user, executing under control of an operating system on an automated processing system, having a context comprising a focus, the operating system selectively providing received user inputs from a human user to a respective user interface element having the focus, comprising:
- determining the context of the operating system executing the at least one software application, based on at least an analysis of the set of dynamically changing user interface elements of the native interface of the computer software application;
defining, through the operating system, a supplemental interface for interaction between the human user and an automated user interface system with the at least one computer software application, selected from a plurality of available supplemental interfaces, each respective supplemental interface supporting at least one human user input unsupported by the respective changing user interface element available in the native interface in the context, a respective supplemental interface being selectively made available through the automated user interface system for the respective user interface element having the focus in dependence on the determined context comprising the focus; and
communicating with the computer software application having focus using at least the input unsupported by the native interface of the respective defined supplemental interface,wherein the computer software application receives input from the respective defined supplemental interface using at least the input unsupported by the native interface without requiring either access to a source code of the computer software application, or modification to the computer software application,wherein the at least one input unsupported by the native interface comprises a speech input corresponding to a graphic user interface command of the native interface.
2 Assignments
0 Petitions
Accused Products
Abstract
In an effort to customize or enhance software applications, configuration data is often used. Configuration settings that are editable by users need not to be limited to a simple flat entry that can be taken out of context anymore. The present invention allows for multiple-levels of configuration settings to interact with each other, so that a single configuration for a given context to be calculated dynamically. In the process, the user gains flexibility to specify more adequately a required change or customization while propagating the information with minimal effort and not requiring additional coding. Furthermore, to simplify a speaker'"'"'s interactions for controlling an automated device, the addition of a superposed layer over graphic user interface may be used. The superposed layer may display coordinates that a speaker may use to navigate the graphic user interface, for example to associate a location with a keyword or a coordinate.
-
Citations
20 Claims
-
1. A method for interaction with a computer software application having a native interface having a set of dynamically changing user interface elements for interaction with a human user, executing under control of an operating system on an automated processing system, having a context comprising a focus, the operating system selectively providing received user inputs from a human user to a respective user interface element having the focus, comprising:
-
determining the context of the operating system executing the at least one software application, based on at least an analysis of the set of dynamically changing user interface elements of the native interface of the computer software application; defining, through the operating system, a supplemental interface for interaction between the human user and an automated user interface system with the at least one computer software application, selected from a plurality of available supplemental interfaces, each respective supplemental interface supporting at least one human user input unsupported by the respective changing user interface element available in the native interface in the context, a respective supplemental interface being selectively made available through the automated user interface system for the respective user interface element having the focus in dependence on the determined context comprising the focus; and communicating with the computer software application having focus using at least the input unsupported by the native interface of the respective defined supplemental interface, wherein the computer software application receives input from the respective defined supplemental interface using at least the input unsupported by the native interface without requiring either access to a source code of the computer software application, or modification to the computer software application, wherein the at least one input unsupported by the native interface comprises a speech input corresponding to a graphic user interface command of the native interface. - View Dependent Claims (2, 3, 4, 5, 6, 7, 18)
-
-
8. A system providing a modified interaction of a human user with a computer software application having a native interface, executing under an operating system having a determinable context comprising at least a dynamically changing state of a human user interface comprising a focus representing a user interface element which receives human user input, the system comprising:
-
a plurality of supplemental interfaces for interaction by the human user with the at least one computer software application, each supplemental interface supporting at least one human user input type which is unsupported by a respective associated human user interface element of the native interface; and a processor configured to select at least one of the plurality of supplemental interfaces based on at least the context which is appropriate for the human user interface element which has the focus, and to communicate in relation to the computer software application through the at least one of the plurality of supplemental interfaces using at least the at least one input type which is supported by the selected supplemental interface and unsupported by the native interface, wherein a human user interaction with the native interface is modified by use of the at least one of the plurality of supplemental interfaces to permit use of the at least one human user input type which is unsupported by the native interface without requiring either access to the source code of the computer software application, or modification to the computer software application, wherein the native interface specifies receipt of human user input through a graphic user interface as a pointing device input or a keyboard input, and at least one of the plurality of supplemental interfaces is configured to receive a speech input and to present information selectively corresponding to the speech input to the native input as a command. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15)
-
-
16. A graphic user interface computer system, communicating with a display device configured to present to a human user a set of graphic objects defined by at least one software application executing under control of an operating system having a context comprising a focus which dynamically associates a human user input with a respective graphic object of the set of graphic objects, having a native graphic user interface, comprising:
-
a speech recognition engine executing under the operating system, configured to receive user speech commands and to output at least one control signal for the at least one software application; a voice controlled graphic layer executing under the operating system, superposed on, or blended over, the native graphic user interface or a representation of the graphic user interface of the at least one software application, the voice controlled graphic layer supporting at least one human user input unsupported by the native graphic user interface, the voice controlled graphic layer being defined independently of the source code of the at least one software application, configured to present a graphic indication to the human user of available speech commands for the respective graphic object of the set of graphic objects of the at least one software application selectively in response to a determination of which graphic object has focus, wherein the superposed voice controlled graphic layer does not disrupt a clarity of the native graphic user interface; and an input port configured to receive a speech input to the speech recognition engine.
-
-
17. A method of speech enabling a software application defining a native graphic user interface having a plurality of distinct graphic user interface events, for execution under a computer operating system, within an automated processing system, having a context defined by at least a graphic user interface status of the computer operating system and any applications executing under the computer operating system, the operating system having a focus for controlling a respective graphic human user interface element which is associated with a human user input, comprising:
-
generating a native graphic user interface of the software application; overlaying, by a supplemental application distinct from the software application, executing under the computer operating system, a plurality of identified regions on the native graphic user interface in dependence on the context and independent of a source code of the software application; receiving a speech input from a human user, identifying a selected identified region, corresponding to a graphic human user interface element of the native graphic user interface of the software application, to grant focus to the corresponding graphic human user interface element if it does not have focus; and generating, selectively in dependence on an analyzed content of the speech input, at least one graphic user interface event supported by the native graphic user interface associated with the graphic human user interface element corresponding to the selected identified region. - View Dependent Claims (19, 20)
-
Specification