Use of a digital assistant in communications

• US 9,462,112 B2
• Filed: 07/01/2014
• Issued: 10/04/2016
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Woolsey, Kerry, Jin, Larry, Halvorsen, Pat, Chory, Susan, Hawkins, Rylan
• Primary Examiner(s): SMITH, CREIGHTON H
• Application Number:
  US14/321,604
• Publication Number:
  US 20150373183A1
• Time in Patent Office: 826 Days
• Field of Search: 348/14.08
• US Class Current: 1/1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72433 for voice messaging, e.g. dictaphones for answering incoming calls H04M1/64
• CPC Class Codes: H04M 1/72448 with means for adapting the functionality of the device according to specific conditions
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2203/357 Autocues for dialog assistance
• CPC Class Codes: H04M 3/527 Centralised call answering arrangements not requiring operator intervention
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

A digital assistant operating on a device is configured to be engaged as an active participant in communications between local and remote parties by listening to voice and video calls and participating in messaging sessions. The digital assistant typically can be initiated by voice using a key word or phrase and then be requested to perform tasks, provide information and services, etc. using voice or gestures. The digital assistant can respond to the request and take appropriate actions. In voice and video calls, the interactions with the digital assistant (i.e., the request, response, and actions) can be heard by both parties to the call as if the digital assistant was a third party on the call. In a messaging session, messages are generated and displayed to each participant so that they can see the interactions with the digital assistant as if it was a participant.

Claims

1. A device, comprising:

one or more processors;

a display that supports a user interface (UI) for interacting with a user of the device; and

a memory device storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

listening in on an audio portion of a video call between local and remote parties,

entering a listening mode by which listening to speech of the local party in the audio portion subsequent to a key word or key phrase being spoken is enabled,

determining an action that is responsive to the speech, the determining including locating applicable context and utilizing the located applicable context,

making an announcement of the determined action by injecting the announcement into the audio portion of the video call so that both the local and remote parties can hear the announcement,

taking the determined action,

returning to the listening mode; and

generating an overlay that is included in an outgoing video stream from the device, the overlay including a representation of interactions between the local party and a digital assistant or the overlay including a representation of status of the digital assistant.

2. The device of claim 1 further including animating at least a portion of the overlay.

3. The device of claim 1 further including initiating the listening mode by using one of voice command, gesture, or manipulation of a virtual or physical control exposed by the device.

4. A device, comprising:

one or more processors;

a display that supports a user interface (UI) for interacting with a user of the device; and

a memory device storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

listening in on an audio portion of a video call between local and remote parties,

entering a listening mode by which listening to speech of the local party in the audio portion subsequent to a key word or key phrase being spoken is enabled,

determining an action that is responsive to the speech, the determining including locating applicable context and utilizing the located applicable context,

making an announcement of the determined action by injecting the announcement into the audio portion of the video call so that both the local and remote parties can hear the announcement,

taking the determined action,

returning to the listening mode, and

naming parties in the video call to whom the determined action is applicable.

5. A device, comprising:

one or more processors;

a display that supports a user interface (UI) for interacting with a user of the device; and

a memory device storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

listening in on an audio portion of a video call between local and remote parties,

entering a listening mode by which listening to speech of the local party in the audio portion subsequent to a key word or key phrase being spoken is enabled,

determining an action that is responsive to the speech, the determining including locating applicable context and utilizing the located applicable context, using data provided by a remote service when making the action determination, or the action determination being made at least in part by an external service that operates substantially remotely from the device,

making an announcement of the determined action by injecting the announcement into the audio portion of the video call so that both the local and remote parties can hear the announcement,

taking the determined action, and

returning to the listening mode.