Semantic canvas

• US 7,640,162 B2
• Filed: 12/14/2004
• Issued: 12/29/2009
• Est. Priority Date: 12/14/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dawson, David, Ickman, Steven W., Ramsey, William D., Laird-McConnell, Thomas M., McConnell, Christopher C., Barklund, Jonas
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): Baker; Matthew H
• Application Number:
  US11/011,811
• Publication Number:
  US 20060129379A1
• Time in Patent Office: 1,841 Days
• Field of Search: 704/9, 704/8, 704/7, 704/257, 704/270.1, 704/275
• US Class Current: 704/270
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

A system for synchronizing a natural language input element and a graphical user interface has a synchronization engine. The synchronization engine is adapted to monitor user interactions with the natural language input element and the graphical user interface and to synchronize the natural language input element and the graphical user interface responsive to the user interactions.

Claims

1. A system for synchronizing a natural language input element and a graphical user interface element, the system comprising:

the natural language input element, wherein the natural language input element displays, via a display device, an indication of a user natural language input on a user interface, wherein the displayed indication of the user natural language input is a representation of words included within an initial natural language input received from a user, and wherein the displayed indication of the user natural language input is displayed in an editable format that allows the user to modify the representation of words included within the initial natural language input by adding words to or removing words from the representation of words included within the initial natural language input;

the graphical user interface element, wherein the graphical user interface element displays, via a display device, on the user interface simultaneously with the display of the indication of the user natural language input, an indication of a user graphical interface input, wherein the displayed indication of the user graphical interface input is an automatically generated control box representation of the initial natural language input received from the user, the displayed indication of the user graphical interface input and the displayed indication of the user natural language input being two different representations of the same initial natural language input received from the user, and wherein the displayed indication of the user graphical interface input is an editable format that allows the user to modify the automatically generated control box representation of words included within the initial natural language input;

a synchronization engine that monitors, via a processor, user interactions with the natural language input element and the graphical user interface element and automatically synchronizes the natural language input element and the graphical user interface element responsive to the user interactions, wherein the synchronization engine automatically synchronizes the natural language input element and the graphical user interface element responsive to the user interactions by automatically changing the displayed indication of the user graphical interface input so as to automatically modify the automatically generated control box representation so as to alter the automatically generated control box representation to incorporate revised logic that is consistent with an edit made to the representation of words included within the initial natural language input, the edit being a word added to or removed from the representation of words included within the initial natural language input; and

a restatement engine that monitors user interactions with the graphical user interface element and utilizes a computer processor that is a component of a computing device to automatically compose a natural language input representative of a modification to the automatically generated control box representation of the initial natural language input received from the user, wherein the modification alters the logic of the displayed representation, and wherein the synchronization engine updates the natural language element so as to automatically respond to the modification to the automatically generated representation by automatically substituting a display of the automatically composed natural language input for the displayed representation of words included within the initial natural language input received from the user, wherein the automatically composed natural language input is an automatically generated re-statement of the displayed representation of words included within the initial natural language input received from the user with modifications being automatically made to the displayed representation of words based directly on the modification to the displayed representation.

2. The system of claim 1, wherein the system further comprises:

a semantic engine that generates a mapping of components of the representation of words included within the initial natural language input to components of the automatically generated representation of the initial natural language input, wherein said step of automatically synchronizing further comprises automatically modifying the automatically generated control box representation based at least in part on references to the mapping.

3. The system of claim 1, wherein the system further comprises:

an execution engine that performs an operation on a data store, the scope of the operation being defined at least in part upon the automatically generated re-statement of the displayed representation of words included within the initial natural language input received from the user.

4. The system of claim 3, wherein the operation comprises a query, and wherein the execution engine retrieves and displays a list of results from a collection of stored data, the scope of the list of results being determined, by a query processing engine, to be in line with the automatically generated re-statement of the displayed representation of words included within the initial natural language input received from the user.

5. The system of claim 1, wherein when the synchronization engine performs said step of synchronizing the natural language input element responsive to user interactions, said step of automatically changing the displayed indication of the user graphical interface is performed automatically in response to the user interactions.

6. The system of claim 1, wherein the displayed indication of the user graphical interface input is an automatically generated control box representation that includes at least one drop box control containing an entry indicative of an element of the initial natural language input received from the user.

7. The system of claim 1, wherein the displayed indication of the user graphical interface input is an automatically generated control box representation that includes at least one check box control containing an entry indicative of an element of the initial natural language input received from the user.

8. The system of claim 1, wherein the displayed indication of the user graphical interface input is an automatically generated control box representation that includes a series of text boxes that each contain an entry such that, collectively, the text boxes represent multiple elements of the initial natural language input received from the user.

9. The system of claim 1, wherein the displayed indication of the user graphical interface input is an automatically generated control box representation that includes a series of text and check boxes that each contain an entry such that, collectively, the text and check boxes represent multiple elements of the initial natural language input received from the user.