System-initiated speech interaction

• US 8,600,763 B2
• Filed: 06/04/2010
• Issued: 12/03/2013
• Est. Priority Date: 06/04/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Brush, Alice Jane Bernheim, Johns, Paul, Anderson, Jen, Missimer, Connie, Yang, Seung, Ku, Jean
• Primary Examiner(s): AZAD, ABUL K
• Application Number:
  US12/794,068
• Publication Number:
  US 20110301958A1
• Time in Patent Office: 1,278 Days
• Field of Search: 704270-276
• US Class Current: 704/275
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Whenever an event occurs on a computing system which will accept a response from a user of the system, the system automatically determines whether or not to enable speech interaction with the system for the event response. Whenever speech interaction is enabled with the system for the event response, the system provides a notification to the user which informs the user of the event and their options for responding thereto, where these options include responding verbally. Whenever the user responds within a prescribed period of time via a voice command (VC), the system attempts to recognize the VC. Whenever the VC is successfully recognized, the system responds appropriately to the VC.

Claims

1. A computer-implemented process for initiating speech interaction (SI) with a computer, comprising:

using the computer to perform the following process actions;

whenever an event occurs on the computer which will accept a response from a user of the computer, automatically determining whether or not to enable SI with the computer for the event response;

whenever SI is enabled with the computer for the event response, providing a notification to the user which informs the user of the event and their options for responding thereto, wherein said options comprise responding verbally;

whenever the user responds to the notification within a prescribed period of time, and whenever the user response comprises a voice command (VC),

attempting to recognize the VC, and

whenever the VC is successfully recognized,

providing visual feedback to the user that their VC has been accepted,

removing the event notification GUI for the current event from the display device,

whenever the VC is not a snooze command, executing the VC, and

whenever the VC is a snooze command,

disabling the enabled SI with the computer,

waiting an amount of time specified by the snooze command,

again enabling SI with the computer using the VC dictionary associated with the current event, and

again providing a notification to the user which informs the user of the current event and their options for responding thereto.

2. The process of claim 1, wherein the process action of automatically determining whether or not to enable SI with the computer for the event response comprises an action of, whenever the event is sanctioned for SI, enabling SI with the computer using a VC dictionary associated with the event.

3. The process of claim 2, wherein the process action of enabling SI with the computer using a VC dictionary associated with the event comprises the actions of:

whenever SI with the computer is not already enabled for one or more previous events,

initiating a speech recognition (SR) engine whenever said engine is not already initiated,

initiating the VC dictionary associated with the current event, and starting SR using said dictionary; and

whenever SI with the computer is already enabled for one or more previous events, and whenever no VCs in the VC dictionaries associated with said previous events are in the VC dictionary associated with the current event,

initiating the VC dictionary associated with the current event, and

starting SR using said dictionary.

4. The process of claim 2, wherein the VC dictionary associated with the event comprises a prescribed number of VCs which are specifically tailored to the event, and each VC comprises either a single word or a sequence of two or more words that are spoken by the user.

5. The process of claim 4, wherein the process action of providing a notification to the user which informs the user of the event and their options for responding thereto comprises an action of, whenever SI with the computer is not already enabled for one or more previous events, and whenever the current event is sanctioned for SI, displaying an event notification GUI for the current event within a new sector on the display device, wherein SI is enabled for said GUI and said GUI comprises one or more of the VCs in the VC dictionary associated with the current event.

6. The process of claim 5, further comprising the actions of:

whenever the user does not respond to the notification within the prescribed period of time;

removing the event notification GUI for the current event from the display device; and

disabling the enabled SI with the computer.

7. The process of claim 4, wherein the process action of providing a notification to the user which informs the user of the event and their options for responding thereto comprises the actions of:

whenever SI with the computer is already enabled for one or more previous events, and whenever no VCs in the VC dictionaries associated with said previous events are in the VC dictionary associated with the current event, displaying an event notification GUI for the current event within a new sector on the display device, wherein SI is enabled for said GUI, said GUI comprises one or more of the VCs in the VC dictionary associated with the current event, and said GUI shares voice input focus with one or more previous event notification GUIs that are associated with said previous events and that are already displayed in other sectors on the display device; and

whenever SI with the computer is already enabled for one or more previous events, and whenever one or more VCs in a VC dictionary associated with one of said previous events are in the VC dictionary associated with the current event, displaying an event notification GUI for the current event within a new sector on a display device of the computer, wherein SI is disabled for said GUI.

8. The process of claim 2, further comprising an action of, whenever the VC is successfully recognized, disabling the enabled SI with the computer.

9. The process of claim 8, wherein the process action of disabling the enabled SI with the computer comprises the actions of:

stopping speech recognition using the VC dictionary associated with the event; and

whenever an event notification graphical user interface (GUI) is displayed for another event that is sanctioned for SI, and whenever SI is currently disabled for said GUI, enabling SI with the computer using a VC dictionary associated with said other event.

10. The process of claim 2, further comprising the actions of:

whenever the VC is not successfully recognized, continuing to wait the prescribed period of time to receive another user response to the notification; and

whenever the user responds to the notification within the prescribed period of time, and whenever the user response comprises a non-voice command,

responding appropriately to the non-voice command, and

disabling the enabled SI with the computer.

11. The process of claim 2, wherein events that are sanctioned for SI comprise at least one of:

a user notification class of system events, wherein,

said notification class comprises events which are generated automatically by an application program running on the computer, and

whenever the event belongs to said notification class, the prescribed period of time is determined by the application program that generated the event;

or

a user activity class of system events, wherein,

said activity class comprises events which are generated by user activity on the computer, and

whenever the event belongs to said activity class, the prescribed period of time will not terminate until the computer receives a user response to the notification.

12. The process of claim 11, wherein:

the events which are generated automatically by an application program running on the computer comprise at least one of an email toast, or a calendar reminder, or an instant message toast; and

the events which are generated by user activity on the computer comprise at least one of user authoring activity within an authoring application program running on the computer, or user file management activity.

13. The process of claim 2, wherein,

the computer comprises a default set of system events that are sanctioned for SI,

a default VC dictionary is associated with each event in the default set, and

the user performs at least one of the following actions:

sanctioning one or more additional system events for SI by adding said additional events to the default set;

or

de-sanctioning one or more specific system events which are currently sanctioned for SI by removing said specific events from the default set;

or

enabling the recognition of one or more new VCs by adding said new VCs to the default VC dictionary associated with a given event;

or

disabling the recognition of one or more existing VCs by removing said existing VCs from the default VC dictionary associated with a given event.

14. In a computer system having a user interface comprising a display device and a sound input device, a computer-implemented process for initiating speech interaction (SI) with the computer, comprising:

using the computer to perform the following process actions;

displaying an event notification graphical user interface (GUI) within a new sector on the display device, wherein,

the GUI informs the user of an event that occurs on the computer and facilitates the user'"'"'s response to the event,

whenever the event is sanctioned for SI, the new sector comprises a first subsector comprising one or more virtual elements, wherein each virtual element represents a different voice command (VC) in a VC dictionary associated with the event and each of the virtual elements comprises a speech bubble within which is displayed text associated with the VC represented by the virtual element, and

whenever the GUI has voice input focus, each speech bubble is highlighted to glow a visually recognizable color, wherein said glow provides a visual indication to the user that SI with the GUI is enabled and draws the user'"'"'s attention to the VCs they can choose from; and

whenever the GUI has voice input focus, inputting a command spoken by the user into the sound input device in response to the GUI.

15. The process of claim 14, wherein the user interface further comprises a GUI selection device and the new sector further comprises at least one of:

a second subsector comprising one or more information fields that identify the event to the user; and

a third subsector comprising one or more manually-selectable (MS) virtual elements, wherein,

each of the MS virtual elements corresponds to a different speech bubble and provides the user with a way to activate the VC represented by its corresponding speech bubble using the GUI selection device,

within each of the MS virtual elements is displayed the same text that is displayed within its corresponding speech bubble, and

each of the speech bubbles comprises a shape which is different than the shape of any of the MS virtual elements.

16. The process of claim 14, wherein,

the user interface further comprises a GUI selection device,

each of the virtual elements comprises a manually-selectable (MS) element within which is displayed text associated with the VC represented by the virtual element,

each MS element provides the user with a way to activate the VC whose text is displayed there-within using the GUI selection device, and

the process action of displaying an event notification GUI comprises an action of, whenever the GUI has voice input focus, augmenting each MS element in a visually recognizable manner, wherein said augmentation provides a visual indication to the user that SI with the GUI is enabled and draws the user'"'"'s attention to the VCs they can choose from, and said augmentation comprises one of the following actions:

displaying a visually recognizable border around the MS element;

or

highlighting the MS element to glow a visually recognizable color;

or

both displaying a visually recognizable border around the MS element and highlighting the MS element to glow a visually recognizable color.

17. The process of claim 14, further comprising an action of, whenever the command spoken by the user is successfully recognized by the computer, lighting up the virtual element that represents said recognized command for a short period of time in order to provide visual feedback to the user that their spoken command has been accepted.

18. A computer-implemented process for initiating speech interaction (SI) with a computer, comprising:

using the computer to perform the following process actions;

whenever an event occurs on the computer which will accept a response from a user of the computer, automatically determining whether or not to enable SI with the computer for the event response, said automatic determination comprising,

whenever the event is sanctioned for SI, and whenever SI with the computer is not already enabled for one or more previous events,

initiating a speech recognition (SR) engine whenever said engine is not already initiated,

initiating a voice command (VC) dictionary associated with the current event, and

starting SR using said dictionary, and

whenever the event is sanctioned for SI, and whenever SI with the computer is already enabled for one or more previous events, and whenever no VCs in the VC dictionaries associated with said previous events are in the VC dictionary associated with the current event,

initiating the VC dictionary associated with the current event, and

starting SR using said dictionary;

providing a notification to the user which informs the user of the current event and their options for responding thereto, said providing comprising,

whenever SI with the computer is not already enabled for one or more previous events, and whenever the current event is sanctioned for SI, displaying an event notification graphical user interface (GUI) for the current event within a new sector on a display device of the computer, wherein SI is enabled for said GUI and said GUI comprises each VC in the VC dictionary associated with the current event, and

whenever SI with the computer is already enabled for one of more previous events, and whenever no VCs in the VC dictionaries associated with said previous events are in the VC dictionary associated with the current event, displaying an event notification GUI for the current event within a new sector on the display device, wherein SI is enabled for said GUI, said GUI comprises one or more of the VCs in the VC dictionary associated with the current event, and said GUI shares voice input focus with one or more previous event notification GUIs that are associated with said previous events and that are already displayed in other sectors on the display device;

whenever the user responds to the notification within a prescribed period of time, and whenever the user response comprises a VC, attempting to recognize the VC using the SR engine; and

whenever the VC is successfully recognized,

providing visual feedback to the user that their VC has been accepted,

removing the event notification GUI for the current event from the display device,

whenever the VC is not a snooze command, executing the VC,

stopping SR using the VC dictionary associated with the current event, and

whenever an event notification GUI is displayed for another event that is sanctioned for SI, and whenever SI is currently disabled for said GUI, initiating a VC dictionary associated with said other event, and starting SR using said dictionary.