Use of a digital assistant in communications

• US 10,135,965 B2
• Filed: 08/31/2016
• Issued: 11/20/2018
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Woolsey, Kerry, Jin, Larry, Halvorsen, Pat, Chory, Susan, Hawkins, Rylan
• Primary Examiner(s): Smith, Creighton
• Application Number:
  US15/253,203
• Publication Number:
  US 20160373571A1
• Time in Patent Office: 811 Days
• Field of Search: None
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 1/72433 for voice messaging, e.g. dictaphones for answering incoming calls H04M1/64
• CPC Class Codes: H04M 1/72448 with means for adapting the functionality of the device according to specific conditions
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2203/357 Autocues for dialog assistance
• CPC Class Codes: H04M 3/527 Centralised call answering arrangements not requiring operator intervention
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

A digital assistant operating on a device is configured to be engaged as an active participant in communications between local and remote parties by listening to voice and video calls and participating in messaging sessions. The digital assistant typically can be initiated by voice using a key word or phrase and then be requested to perform tasks, provide information and services, etc. using voice or gestures. The digital assistant can respond to the request and take appropriate actions. In voice and video calls, the interactions with the digital assistant (i.e., the request, response, and actions) can be heard by both parties to the call as if the digital assistant was a third party on the call. In a messaging session, messages are generated and displayed to each participant so that they can see the interactions with the digital assistant as if it was a participant.

Claims

1. A device, comprising:

one or more processors;

a display that supports a user interface (UI) for interacting with a user of the device; and

a memory device storing computer-readable instructions which, when executed by the one or more processors, cause the device to;

during an active communication session between the user and a remote party distinct from a digital assistant, configure the digital assistant to interact with the user with text messages supported by a messaging application that is operable on the device, wherein the digital assistant reads text messages sent by the user to the digital assistant, and the digital assistant sends text messages to the user that appear on the UI for the messaging application,

receive an input communication from the user, in which the received input is a request for the digital assistant to perform a task or gather information,

transmit a message to the remote party after the user inputs the communication to the digital assistant, such that the transmitted message to the remote party provides details about the user'"'"'s input communication,

determine an action that is responsive to the text messages read by the digital assistant during the active communication session between the user and the remote party, the determining including locating applicable context and utilizing the located applicable context, and

send a text message through the messaging application to the user responsively to the action determination.

2. The device of claim 1 in which the computer-readable instructions, when executed by the one or more processors, further cause the action determination to use data provided by an external service that operates substantially remotely from the device, or the action determination being made at least in part by the external service.

3. The device of claim 1 in which the computer-readable instructions, when executed by the one or more processors, further cause the device to enable the digital assistant to request confirmation in a text message from the user prior to performing an action.

4. The device of claim 1 in which the action comprises one or more of finding information, answering a question, making a purchase, making a reservation, sharing contact information, scheduling a meeting, sending a message, looking at a user'"'"'s calendar, scheduling a reminder, playing games, taking notes, scheduling an alarm or wakeup reminder, adding a task to a task list, interacting with a search service, showing or sharing emails, showing or sharing files, sending a link to a website, or sending a link to a resource.

5. The device of claim 1 in which the computer-readable instructions, when executed by the one or more processors, further cause the device to enable the determined action to be performed.

6. The device of claim 5 in which the digital assistant performs the determined action in whole or part.

7. The device of claim 1 in which the sent text message conveys a result of a performed action.

8. A method by which a digital assistant operating on a local computing device interacts in a messaging session supported by a messaging application, comprising:

configuring the digital assistant to receive messages from a local party or one or more remote parties to the messaging session using the messaging application so that the digital assistant appears on a messaging application user interface (UI) as another party to the messaging session, wherein the messaging session includes the local party and the one or more remote parties;

reading the messages from the local party or the one or more remote parties in the messaging session that are sent to the digital assistant, wherein the messaging session is an active communication session between the local and the one or more remote parties;

determining an action that is responsive to the read messages during the active communication session between the local and the one or more remote parties, the determining including locating applicable context and utilizing the located applicable context, and using data provided by an external service that operates substantially remotely from the device, or the action determination being made at least in part by the external service; and

transmitting a message to the remote party when the digital assistant receives an input communication from the local party, such that the transmitted message to the remote party provides details about the input communication received at the digital assistant from the local party.

9. The method of claim 8 further comprising sending a message to the local party or the one or more remote parties in the messaging session, the sent message being displayed on the UI as originating from the digital assistant.

10. The method of claim 9 in which the sent message includes information pertaining to the determined action.

11. The method of claim 10 in which the determined action comprises collecting information and the sent message provides the collected information.

12. The method of claim 10 in which the determined action comprises finding an answer to a question and the sent message provides the found answer.

13. The method of claim 8 further comprising configuring the digital assistant to be invoked by the local party or the one or more remote parties during the messaging session using one of message, voice command, gesture, or manipulation of a virtual or physical control exposed by the device.

14. The method of claim 8 further comprising storing messages to create a record of interactions between the local party and the digital assistant.

15. One or more computer-readable memory devices in an electronic device storing computer-readable instructions, which when executed by one or more processors cause the electronic device to:

expose a digital assistant to messages communicated between a local party using the electronic device and one or more remote parties using remote computing devices through a messaging application, the one or more remote parties being distinct entities from the digital assistant, wherein the digital assistant can read the messages, and the local party or the one or more remote parties can communicate with the digital assistant using a message from the messaging application;

determine an action that is responsive to a message directed to and received at the digital assistant which originates from the local party, the determining including locating applicable context and utilizing the located applicable context; and

transmit a corresponding message to the remote party after the digital assistant receives the message from the local party, such that the transmitted message to the remote party provides details about the received message at the digital assistant from the local party.

16. The one or more computer-readable memory devices of claim 15 in which the determined action comprises one or more of providing information, answering a question, performing a task, or providing a link to a website or remote resource.

17. The one or more computer-readable memory devices of claim 15 in which the computer-readable instructions further cause the electronic device to associate the digital assistant with the action in a displayed message as if the digital assistant is another party to the messaging session.

18. The one or more computer-readable memory devices of claim 15 in which the computer-readable instructions further cause the electronic device to perform the determined action.

19. The one or more computer-readable memory devices of claim 18 in which the digital assistant receives an input from the local party or the one or more remote parties prior to performing the determined action.