Multiple shell multi faceted graphical user interface

• US 8,595,642 B1
• Filed: 09/30/2008
• Issued: 11/26/2013
• Est. Priority Date: 10/04/2007
• Status: Expired due to Fees

• Current Assignee: Great Northern Research, LLC
• Original Assignee: Great Northern Research, LLC
• Inventors: Lagassey, Paul J.
• Primary Examiner(s): Nunez, Jordany
• Application Number:
  US12/241,035
• Time in Patent Office: 1,883 Days
• Field of Search: 715/781, 715/778, 704/275
• US Class Current: 715/781
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A multi faceted graphic user interface with multiple shells or layers may be provided for interaction with a user to speech enable interaction with applications and processes that do not necessarily have native support for speech input. The shells may be components of an operating system or of a parent application which supports such shells. Each shell has multiple facets for displaying applications and processes, and typically speech and other input is directed the application or process in the facet which has focus within the active shell. These multiple shells lend themselves to grouping of input or grouping of related applications and processes. For example, input from a speech recognizer, a mouse and a keyboard may each be directed at different shells; or a user may group related windows within various shells, such that all documents are displayed in one shell and all windows of an instant messaging application are displayed in another, thereby enabling better organization of work and work flow.

Claims

1. A computing system, comprising:

a processor, random access memory, and nonvolatile storage;

an operating system, stored in the non-volatile storage, executing on the processor, and employing the random access memory for operational tasks;

a plurality of software shells executing within the operating system, each software shell communicating with the operating system;

a plurality of facets, at least two facets executing within at least two different software shells;

each facet supporting execution of a plurality of applications thereunder;

a pointing input device providing a pointing device input to the operating system;

a natural language speech input user interface device providing at least a natural language speech input comprising natural language speech input commands and data to the operating system; and

wherein the operating system pipes the pointing device input to a first application under a first facet executing within a first shell having a pointing device focus;

wherein the operating system pipes the natural language speech input to a second application under a second facet executing within a second shell having a speech focus;

wherein speech focus is different from the pointing device focus, and the speech focus and the pointing device focus are concurrently available to a single user;

the plurality of software shells being concurrently displayed to the single user through a user interface;

wherein the speech input is analyzed in a context-sensitive manner for a defined action which selectively alters the speech focus from the second application to a respective third application under a third facet executing within a third software shell;

wherein the plurality of software shells share a common focus, such that a single user natural language speech input is concurrently operated upon by a plurality of applications under different facets executing within different software shells; and

wherein at least one application under at least one facet executing within at least one shell may communicate with a different application under a different facet executing in at least one other shell.

2. The computing system according to claim 1, wherein the at least one application comprises a telephone dialer application.

3. The computing system according to claim 1, wherein at least one application is configured to receive the speech input as a dictation input.

4. The computing system according to claim 1, wherein each shell has at least two facets, each facet capable of displaying either an application or a process, and wherein the pointing device user input grants focus to the application displaying within one of said at least two facets.

5. A method of computing, comprising:

providing a processor, random access memory, and nonvolatile storage;

executing an operating system, stored in the non-volatile storage, on the processor, and employing the random access memory for operational tasks;

a plurality of software shells executing within the operating system, each software shell communicating with the operating system;

providing a plurality of executing facets, at least two facets executing within at least two different software shells, each facet supporting execution of a plurality of applications thereunder;

receiving a pointing input device input to the operating system;

receiving natural language speech from a user interface device, providing at least a natural language speech input comprising natural language speech input commands and data to the operating system; and

piping the pointing device input by the operating system to a first application under a first facet executing within a first shell having a pointing device focus;

piping the natural language speech input by the operating system to a second application under a second facet executing within a second shell having a speech focus;

wherein speech focus is different from the pointing device focus, and the speech focus and the pointing device focus are concurrently available to a single user;

concurrently displaying the plurality of software shells to the single user through a user interface;

analyzing the speech input in a context-sensitive manner for a defined action which selectively alters the speech focus from the second application to a respective third application under a third facet executing within a third software shell;

sharing a common focus for the plurality of software shells, such that a single user natural language speech input is concurrently operated upon by a plurality of applications under different facets executing within different software shells; and

executing at least one application under at least one facet within at least one shell and communicating with a different application under a different facet executing in at least one other shell.

6. The method according to claim 5, wherein the at least one application comprises a telephone dialer application.

7. The method according to claim 5, wherein at least one application receives the speech input as a dictation input.

8. The method according to claim 5, wherein each shell has at least two facets, each facet capable of displaying either an application or a process, and wherein the pointing device user input grants focus to the application displaying within one of said at least two facets.

9. The method according to claim 5, wherein each respective facet is adapted to automatically change at least one of a size, a shape and a position in response to a state of other respective facets within the same shell.

10. The method according to claim 5, wherein a number and size of each respective facet is predetermined.

11. The method according to claim 5, wherein the predetermined number and size of each respective facet differs between respective ones of the plurality of shells.

12. The method according to claim 5, wherein the plurality of shells are supplied to the operating system by a separate software construct.

13. The method according to claim 5, wherein the applications which occupy each respective facet are determined by the user.

14. The method according to claim 5, wherein the applications which occupy each respective facet are determined by at least one command received through the natural language speech input.

15. The method according to claim 5, wherein the applications which occupy each respective facet are determined by the operating system.

16. The method according to claim 5, wherein at least one of a size, a shape, and a position of a facet is dynamically varied in dependence on information displayed in the facet.

17. A computer readable medium storing therein non-transitory instructions for controlling a computer system comprising a processor, random access memory, and nonvolatile storage to perform the method comprising:

executing an operating system, stored in the non-volatile storage, on the processor, and employing the random access memory for operational tasks;

a plurality of software shells executing within the operating system, each software shell communicating with the operating system;

providing a plurality of executing facets, at least two facets executing within at least two different software shells, each facet supporting execution of a plurality of applications thereunder;

receiving a pointing input device input to the operating system;

receiving natural language speech from a user interface device, providing at least a natural language speech input comprising natural language speech input commands and data to the operating system; and

piping the pointing device input by the operating system to a first application under a first facet executing within a first shell having a pointing device focus;

piping the natural language speech input by the operating system to a second application under a second facet executing within a second shell having a speech focus;

wherein speech focus is different from the pointing device focus, and the speech focus and the pointing device focus are concurrently available to a single user;

concurrently displaying the plurality of software shells to the single user through a user interface;

analyzing the speech input in a context-sensitive manner for a defined action which selectively alters the speech focus from the second application to a respective third application under a third facet executing within a third software shell;

sharing a common focus for the plurality of software shells, such that a single user natural language speech input is concurrently operated upon by a plurality of applications under different facets executing within different software shells; and

executing at least one application under at least one facet within at least one shell and communicating with a different application under a different facet executing in at least one other shell.

18. The computer readable medium according to claim 17, wherein the at least one application comprises a telephone dialer application.

19. The computer readable medium according to claim 17, wherein at least one application receives the speech input as a dictation input.

20. The computer readable medium according to claim 17, wherein each shell has at least two facets, each facet capable of displaying either an application or a process, and wherein the pointing device user input grants focus to the application displaying within one of said at least two facets.

21. The computer readable medium according to claim 17, wherein each respective facet is adapted to automatically change at least one of a size, a shape and a position in response to a state of other respective facets within the same shell.