Multimodal input using scratchpad graphical user interface to edit speech text input with keyboard input

• US 9,081,590 B2
• Filed: 06/24/2008
• Issued: 07/14/2015
• Est. Priority Date: 06/24/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Brown, Eric C., Mowatt, David
• Primary Examiner(s): Godbold, Douglas
• Assistant Examiner(s): Villena, Mark
• Application Number:
  US12/145,487
• Publication Number:
  US 20090319266A1
• Time in Patent Office: 2,576 Days
• Field of Search: 345/173, 715/780, 719/318
• US Class Current: 1/1
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A system and method for multimodal input into an application program. The method may include performing speech recognition on speech audio input to thereby produce recognized speech text input for insertion into a document of an application program, the document having keyboard focus. The method may also include identifying the document as being text service framework unaware. The method may further include displaying the recognized speech text input in a scratchpad graphical user interface for editing the recognized speech text input. The method may further include reflecting keyboard input bound for the document to the scratchpad graphical user interface, while preserving the keyboard focus of the document. The method may also include displaying the reflected keyboard input on the scratchpad graphical user interface, to thereby effect edits in the recognized speech text input.

Claims

1. A multimodal input system for use with an application program executed on a computing device, the system comprising:

a dictation module configured to receive a recognized speech text input for insertion into a document that has keyboard focus, the recognized speech text input including text recognized from speech audio input by a speech recognition module;

a scratchpad graphical user interface (GUI) configured to display the recognized speech text input, to receive keyboard input to effect edits in the recognized speech text input without changing a keyboard focus from the document in the application program, and to generate an edited speech input for insertion into the document; and

a document-specific keyboard handler including a reflector configured to selectively reflect keyboard input, received from a foreground queue and bound for the document, to the scratchpad GUI for editing of the recognized speech text input, wherein the reflector includes a filter that is configured to programmatically filter the keyboard input while keyboard input is being reflected, and to refrain from reflecting exempted keyboard input that is determined to match a predetermined filter parameter, and instead pass the exempted keyboard input to the document.

2. The multimodal input system of claim 1,

wherein the dictation module is further configured to receive a notification from a document-specific text service module of a text services framework, prior to receiving the recognized speech text input, which indicates that the document with which the document-specific text service module is associated has keyboard focus, and also which includes a first document identifier that identifies the document in focus.

3. The multimodal input system of claim 2,

wherein the document-specific text service module is configured to store a local copy of the first document identifier;

and

wherein the dictation module is further configured to set a local keyboard focus by changing a value of the local copy of the first document identifier to be a second document identifier that is associated with the scratchpad GUI, thus informing the document-specific text service module that a scratchpad document of the scratchpad GUI has keyboard focus, without affecting the keyboard focus registered with other components of the text services framework of an operating system of the computing device.

4. The multimodal input system of claim 2, wherein the dictation module is further configured to send a session control command instructing the document-specific keyboard handler to start or stop reflecting the keyboard input to the dictation module for display on the scratchpad GUI, based on a set of predetermined rules.

5. The multimodal input system of claim 4, wherein the rules include routing the keyboard input to the document if the document is text service framework unaware.

6. The multimodal input system of claim 5, wherein the dictation module is configured to determine that the document is text service framework unaware by referencing the first document identifier associated with the document, and determining whether the document is of a type that is text service framework unaware.

7. The multimodal input system of claim 4, wherein the rules include routing the keyboard input to the document according to a user setting or a user command.

8. The multimodal input system of claim 7, wherein the user command is a voice enabled command received via speech audio input.

9. The multimodal input system of claim 4, wherein the session control command is sent from the dictation module via the document-specific text service module to the document-specific keyboard handler.

10. A method for multimodal input into an application program, comprising:

on a computing device,

performing speech recognition on speech audio input to thereby produce recognized speech text input for insertion into a document of an application program, the document having keyboard focus;

identifying the document as being text service framework unaware;

displaying the recognized speech text input in a scratchpad graphical user interface for editing the recognized speech text input;

selectively reflecting keyboard input bound for the document to the scratchpad graphical user interface to effect edits in the recognized speech text input, while preserving the keyboard focus of the document;

programmatically filtering the keyboard input while keyboard input is being reflected, to refrain from reflecting exempted keyboard input that is determined to match a predetermined filter parameter, and instead pass the exempted keyboard input to the document;

displaying the reflected keyboard input on the scratchpad graphical user interface, to thereby effect edits in the recognized speech text input; and

displaying the exempted keyboard input in the document.

11. The method of claim 10, further comprising, generating an edited speech input based on the recognized speech text input and the edits;

and

inserting the edited speech input into the document with keyboard focus.

12. The method of claim 11, wherein performing speech recognition further includes producing correction information including recorded portions of speech audio input and/or alternate recognition candidate words or phrases for a selected span of speech audio input.

13. The method of claim 12, wherein generating the edited speech input further includes presenting the recorded portions of speech audio input and/or alternate recognition candidate words or phrases to a user during an edit session via the scratchpad graphical user interface, to enable the user to appropriately make the edits in the recognized speech text input.

14. The method of claim 11, further comprising, prior to displaying the recognized speech text input in the scratchpad graphical user interface, applying a set of predetermined rules to determine that an edit session is to be initiated.

15. The method of claim 14, wherein the predetermined rules include determining that a user command or user setting does not specify forgoing the edit session.

16. A method for multimodal input into an application program, comprising:

on a computing device,

receiving recognized speech text input for insertion into a document of an application program, the document being text service framework unaware and having keyboard focus;

displaying the recognized speech text input in a scratchpad graphical user interface for editing the recognized speech text input;

selectively reflecting keyboard input bound for the document to the scratchpad graphical user interface to effect edits in the recognized speech text input, while preserving the keyboard focus of the document; and

programmatically filtering the keyboard input while keyboard input is being reflected, to refrain from reflecting exempted keyboard input that is determined to match a predetermined filter parameter, and instead pass the exempted keyboard input to the document.