Multiple shell multi faceted graphical user interface

• US 9,189,197 B1
• Filed: 05/22/2013
• Issued: 11/17/2015
• Est. Priority Date: 10/04/2007
• Status: Active Grant

• Current Assignee: Great Northern Research, LLC
• Original Assignee: Great Northern Research, LLC
• Inventors: Lagassey, Paul J.
• Primary Examiner(s): Nunez, Jordany
• Application Number:
  US13/900,480
• Time in Patent Office: 909 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A multi faceted graphic user interface with multiple shells or layers may be provided for interaction with a user to speech enable interaction with applications and processes that do not necessarily have native support for speech input. The shells may be components of an operating system or of a parent application which supports such shells. Each shell has multiple facets for displaying applications and processes, and typically speech and other input is directed the application or process in the facet which has focus within the active shell. These multiple shells lend themselves to grouping of input or grouping of related applications and processes. For example, input from a speech recognizer, a mouse and a keyboard may each be directed at different shells; or a user may group related windows within various shells, such that all documents are displayed in one shell and all windows of an instant messaging application are displayed in another, thereby enabling better organization of work and work flow.

Claims

1. A computing system, comprising:

a processor;

a memory configured to store instructions for controlling the processor;

a user interface device configured to receive natural language speech input;

a graphic user interface comprising an array of facets arranged in a tiled configuration, at least one of the facets being associated with at least one application; and

a natural language speech parser configured to parse the received natural language speech input according to instruction grammars to determine instructions and respective parameters of the determined instructions,

wherein, upon receipt of natural language speech input comprising a first instruction and a first parameter of the first instruction, the processor is configured to cause a focus to shift to a facet associated with a first application associated with the respective first parameter of the first instruction and to vary the first facet from a first size to a second size, the second size occupying substantially an entire viewable portion of the graphic user interface, and pass the first instruction and respective first parameter for processing by the first application associated with the facet, and upon subsequent receipt of natural language speech input comprising a second instruction, the processor is configured to cause the focus to shift away from the facet associated with the first application associated with the first parameter and return the first facet to the first size, and

wherein, upon receipt of natural language speech input comprising the first instruction and a second parameter, the processor is configured to cause the focus to shift to a second facet associated with a second application associated with the second parameter of the first instruction, to vary the second facet from a third size to the second size, and upon subsequent receipt of natural language speech input comprising the second instruction, to cause the focus to shift away from the second facet associated with the second parameter and to return the second facet to the third size.

2. The system of claim 1, wherein the first instruction comprises an instruction adapted to open at least one of the first application and the second application.

3. The system of claim 1, wherein the second instruction comprises an instruction adapted to close at least one of the first application and the second application.

4. The system of claim 1, further comprising a video camera system configured to determine a respective facet at which a human user is pointing.

5. A method for controlling a human computer interface, comprising:

providing a user interface device for receiving natural language speech input;

generating a graphic user interface comprising an array of facets arranged in a tiled configuration, at least one of the facets being associated with at least one application; and

implementing a natural language speech parser configured to parse received natural language speech input according to instruction grammars to determine instructions and respective parameters of the determined instructions,

wherein, upon receipt of natural language speech input comprising a first instruction and a first parameter of the first instruction, shifting a focus to a facet associated with a first application associated with the respective first parameter of the first instruction and to vary the first facet from a first size to a second size, the second size occupying substantially an entire viewable portion of the graphic user interface, and passing the first instruction and respective first parameter to the first application associated with the facet for processing, and upon subsequent receipt of natural language speech input comprising a second instruction, shifting a focus away from the facet associated with the first application associated with the first parameter and return the first facet to the first size, and

wherein, upon receipt of natural language speech input comprising the first instruction and a second parameter, shifting the focus to a second facet associated with a second application associated with the second parameter of the first instruction, varying the second facet from a third size to the second size, and upon receipt of natural language speech input comprising the second instruction, shifting the focus from the second facet associated with the second parameter and returning the second facet to the third size.

6. The method of claim 5, wherein the first instruction comprises an instruction adapted to open at least one of the first application and the second application.

7. The method of claim 5, wherein the array of facets represents multiple concurrently executing applications, further comprising causing the facets associated with respective applications to automatically move in and out of a display area of the graphic user interface in accordance with an activity of the respective application.

8. The method of claim 5, wherein the array of facets represents multiple concurrently executing applications, further comprising causing cause the facets associated with respective applications automatically change shape in accordance with an activity of the respective application.

9. The method of claim 5, wherein the focus for natural language speech input is different from a focus for a pointing device input.

10. The method of claim 5, further comprising acquiring a video image to determine a respective facet at which a human user is looking.

11. The method of claim 5, further comprising storing state information of the graphic user interface before shifting a focus to the first application.

12. The method of claim 5, wherein the graphic user interface displays a prompt representing a plurality of possible commands corresponding to the natural language speech input.

13. The method of claim 12, wherein the prompt is context sensitive.

14. The method of claim 5, wherein the at least one application has an associated priority, further comprising processing the natural language speech input in order of application priority.

15. The method of claim 5, further comprising concurrently parsing the natural language speech input according to a set of available grammars representing respectively different commands, and assessing a completeness of the natural language speech input with respect to the set of available grammars.

16. A non-transitory computer readable medium containing instructions for controlling an automated processor to perform the steps, comprising:

generating a graphic user interface comprising an array of facets arranged in a tiled configuration, at least one of the facets being associated with at least one application; and

implementing a natural language speech parser configured to parse received natural language speech input according to instruction grammars to determine instructions and respective parameters of the determined instructions,

wherein, upon receipt of natural language speech input comprising a first instruction and a first parameter of the first instruction, the processor causes a focus to shift to a facet associated with a first application associated with the respective first parameter of the first instruction and to vary the first facet from a first size to a second size, the second size occupying substantially an entire viewable portion of the graphic user interface, and to pass the first instruction and respective first parameter for processing by the first application associated with the facet, and upon subsequent receipt of natural language speech input comprising a second instruction, the processor causes the focus to shift away from the facet associated with the first application associated with the first parameter and return the first facet to the first size, and

wherein, upon receipt of natural language speech input comprising the first instruction and a second parameter, the processor causes the focus to shift to a second facet associated with a second application associated with the second parameter of the first instruction, to vary the second facet from a third size to the second size, and upon receipt of natural language speech input comprising the second instruction, the processor causes the focus to shift from the second facet associated with the second parameter and to return the second facet to the third size.

17. The non-transitory computer readable medium of claim 16, wherein the first instruction comprises an instruction adapted to open at least one of the first application and the second application.

18. The non-transitory computer readable medium of claim 16, wherein the array of facets represents multiple concurrently executing applications, further comprising causing the facets associated with respective applications to automatically move in and out of a display area of the graphic user interface in accordance with an activity of the respective application.

19. The non-transitory computer readable medium of claim 16, wherein the array of facets represents multiple concurrently executing applications, further comprising causing cause the facets associated with respective applications automatically change shape in accordance with an activity of the respective application.

20. The non-transitory computer readable medium of claim 16, further comprising acquiring a video image to determine a respective facet at which a human user is looking.