USE OF A DIGITAL ASSISTANT IN COMMUNICATIONS

• US 20150373183A1
• Filed: 07/01/2014
• Published: 12/24/2015
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Woolsey, Kerry, Jin, Larry, Halvorsen, Pat, Chory, Susan, Hawkins, Rylan
• Granted Patent: US 9,462,112 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72433 for voice messaging, e.g. dictaphones for answering incoming calls H04M1/64
• CPC Class Codes: H04M 1/72448 with means for adapting the functionality of the device according to specific conditions
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2203/357 Autocues for dialog assistance
• CPC Class Codes: H04M 3/527 Centralised call answering arrangements not requiring operator intervention
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

A digital assistant operating on a device is configured to be engaged as an active participant in communications between local and remote parties by listening to voice and video calls and participating in messaging sessions. The digital assistant typically can be initiated by voice using a key word or phrase and then be requested to perform tasks, provide information and services, etc. using voice or gestures. The digital assistant can respond to the request and take appropriate actions. In voice and video calls, the interactions with the digital assistant (i.e., the request, response, and actions) can be heard by both parties to the call as if the digital assistant was a third party on the call. In a messaging session, messages are generated and displayed to each participant so that they can see the interactions with the digital assistant as if it was a participant.

Claims

1. A method for using a digital assistant operating on a device during a voice call carried over a communications network between a local party and a remote party, comprising:

configuring the digital assistant to listen to the voice call between the local party and the remote party;

initiating the digital assistant when the local party says a key word or key phrase;

making a determination of an action to be performed by the digital assistant in view of what the local party says on the voice call after the digital assistant is initiated;

enabling the digital assistant to perform the determined action; and

returning the digital assistant to a listening mode so that the digital assistant is reinitiated when the locals party says the key word or key phrase.

2. The method of claim 1 further including enabling the digital assistant to ask questions in which answers to the questions are used in making the determination on the action to be performed.

3. The method of claim 1 further including configuring the digital assistant for verbally announcing the determined action to the local and remote parties by injecting audio into the voice call.

4. The method of claim 1 further including configuring the digital assistant for visually announcing the determined action to the local party, the visual announcement comprising at least one of graphic object or text object that is displayed on the device.

5. The method of claim 4 in which the visual announcement is configured for display on one of a device associated with the local party or a device associated with the remote party.

6. The method of claim 4 further including using a phone application to display the visual announcement.

7. The method of claim 4 further including animating the graphics in the visual announcement.

8. The method of claim 4 further including showing a status for the digital assistant by animating either the graphic object or text object.

9. The method of claim 1 further including configuring the digital assistant, responsively to the local user'"'"'s speech, for performing at least one of sharing contact information, sharing a stored contact, scheduling a meeting, looking at a user'"'"'s calendar, scheduling a reminder, adding another party to a current call, playing games, making purchases, taking notes, scheduling an alarm or wakeup reminder, changing a current call from a voice call to a video call, changing a current call from a video call to a voice call, reconnecting a dropped call, scheduling a call interruption, interacting with a search service, showing or sharing emails, sharing or showing files, sending a link to a website, or sending a link to a resource.

10. The method of claim 1 further including enabling the digital assistant to request confirmation from the local user prior to performing an action.

11. A device, comprising:

one or more processors;

a display that supports a user interface (UI) for interacting with a user of the device; and

a memory device storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of;

listening in on an audio portion of a video call between local and remote parties,

entering a listening mode by which listening to speech of the local user in the audio portion subsequent to a key word or key phrase being spoken is enabled,

determining an action that is responsive to the speech, the determining including locating applicable context and utilizing the located applicable context,

making an announcement of the determined action by injecting the announcement into the audio portion of the video call so that both the local and remote parties can hear the announcement,

taking the determined action, and

returning to the listening mode.

12. The device of claim 11 in which the context comprises one or more of time/date, locations of either the local user or the remote user, language, schedule, applications installed on the device, user preferences, user behaviors, stored contacts, call history, messaging history, browsing history, device type, device capabilities, communication network type, communication features or functionalities, mobile data plan restrictions, or data associated with other parties to the video call.

13. The device of claim 11 further including generating an overlay that is included in an outgoing video stream from the device, the overlay including a representation of interactions between the local user and a digital assistant or the overlay including a representation of status of the digital assistant.

14. The device of claim 13 further including animating at least a portion of the overlay.

15. The device of claim 13 further including initiating the listening mode by using one of voice command, gesture, or manipulation of a virtual or physical control exposed by the device.

16. The device of claim 11 further including naming parties in the video call to whom the action is applicable.

17. The device of claim 11 in which the applicable context includes video call state.

18. The device of claim 11 further including using data provided by a remote service when making the action determination.

19. The device of claim 11 in which the action determination is made at least in part by an external service that operates substantially remotely from the device.

20. One or more computer-readable storage media storing instructions which, when executed by one or more processors disposed in a device, perform a method for using a digital assistant in messaging communications, the method comprising the steps of:

establishing a messaging session between local and remote users;

listening for voice commands from the local user;

determining an action that is responsive to a voice command from the local user;

generating messages that describe the determined action;

sending a generated message to each of the local and remote users;

performing the determined action; and

iteratively performing the steps of listening, determining, generating, and sending.

21. The one or more computer-readable storage media of claim 20 further including storing messages to create a record of interactions between the local user and a digital assistant.

22. The one or more computer-readable storage media of claim 20 further including applying context when determining the action, the context relating one or more of the local user, a local device, the remote user, a remote device, a plurality or remote users, or a plurality of remote devices.

23. The one or more computer-readable storage media of claim 20 further including interacting with a digital assistant that is instantiated on a device associated with the remote user.