Multi-modal conversational intercom

• US 10,587,708 B2
• Filed: 03/28/2016
• Issued: 03/10/2020
• Est. Priority Date: 03/28/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Laird-McConnell, Thomas Matthew, Cheng, Lili, Lewallen, James H., Cranshaw, Justin, Jensen, Craig, Ickman, Steve, Portnoy, William L., Lu, Qi, Driscoll, Daniel J.
• Primary Examiner(s): Cheema, Umar
• Assistant Examiner(s): Rotolo, Anthony T
• Application Number:
  US15/083,019
• Publication Number:
  US 20170279906A1
• Time in Patent Office: 1,443 Days
• Field of Search: 709202, 709203, 709206
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/066 Format adaptation, e.g. format conversion or compression
• CPC Class Codes: H04L 51/56 Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

The techniques discussed herein allow an agent that provides dialog-based services to converse with a user via any mode. A central service programmed to be able to communicate via any mode collects information from agents necessary to act on the agents'"'"' behalf. In order to ensure that an agent has a maximal presence across modes, the central service provides auxiliary services such as language translation, fidelity upgrading and/or downgrading, modal constraint conformation, etc. Moreover, the central service provides session tracking via user profiles stored at the intercom so that if a user changes a mode by which the user is conversing with an agent, the central service can keep the context of the conversation across the new mode. Furthermore, agents can communicate with other agents via the central service, allowing other agents to provide services to agents or to add their services to an ongoing dialog with a user.

Claims

1. A system comprising:

one or more processors;

a network interface; and

computer-readable media having stored thereon computer-executable instructions, that, when executed, cause the one or more processors to perform operations comprising;

maintaining a plurality of communication channels for electronic communication via the network interface;

accessing a user profile of a user, the user profile being associated with one or more user-communication channels by which a user device of the user is able to communicate;

maintaining a directory of a plurality of agents, each agent being a computer program that, when executed, implements a dialog-based service to provide information in a user interface via an exchange of natural language, the plurality of communication channels being shared by the plurality of agents, each agent being linkable to one or more of the communication channels;

linking one or more first-agent communication channels to a first agent from the plurality of agents, wherein linking the one or more first-agent communication channels to the first agent comprises causing an intercom to use data stored in an agent directory to make requests to send or receive messages or files over the one or more first-agent communication channels on behalf of the first agent;

receiving a natural language communication from the user of the user device via the network interface through one of the user-communication channels, the natural language communication causing a service trigger to activate the first agent for communication with the user device in response to identifying relevant services specified in the natural language communication;

based at least in part on the service trigger, creating a dialog session between the first agent and the user device via the network interface using one or more of the user-communication channels and the first-agent communication channels;

capturing dialog between the first agent and the user device via the dialog session;

detecting a change of activity on a first communication channel;

detecting a change of activity on a second communication channel of the user-communication channels; and

maintaining the dialog session between the first agent and the user using the second communication channel, wherein maintaining the dialog session includes continuing the dialog between the first agent and the user and routing messages via an API endpoint associated with the first agent over the second communication channel, the second communication channel being one of the first-agent communication channels.

2. The system of claim 1, wherein the directory of a plurality of agents further comprises references to application programming interface (API) endpoints associated with the plurality of agents, wherein creating the dialog session includes:

receiving messages from the user over one of the user-communication channels;

relaying the messages to the first agent via an API endpoint associated with the first agent;

receiving responses from the first agent via the API endpoint; and

relaying the responses to the user device over one of the user-communication channels on behalf of the first agent.

3. The system of claim 1, wherein the detecting the change of activity includes:

monitoring or receiving one or more of an active status of the second communication channel;

a user indication;

a preferred communication channel;

a communication channel from which a majority of communication is being received;

a communication channel from which most-recent communication has been received;

or a communication channel preferred to be a channel-in-use based at least in part on one or more of;

a time, a location of the user, an IP address associated with communication received from the communication channel, a calendar event, or a status of a device.

4. The system of claim 1, the operations further comprising formatting communication between the first agent and the user device to conform to standards of the communication channel.

5. The system of claim 1, wherein the first agent is executed at a location remote from the system and communicates with the system via the network interface.

6. The system of claim 1, wherein creating a dialog further comprises:

selecting the first agent based at least in part on a reference to services provided by the first agent that correspond to keywords associated with the service trigger.

7. The system of claim 1, wherein the service trigger comprises a first service trigger, and the operations further comprise:

identifying from the dialog session a second service trigger associated with a service provided by a second agent of the plurality of agents; and

adding the second agent to the dialog session, the adding including;

relaying, to the second agent via an API endpoint associated with the second agent, messages sent to and from the first agent and the user device, respectively; and

relaying messages from the second agent via the API endpoint associated with the second agent to the first agent and the user device over the communication channel.

8. The system of claim 1, the operations further comprising:

receiving an indication from the first agent that a request received via the dialog session from the communication channel is outside a scope of services provided by the first agent;

identifying a second agent that provides services associated with the request; and

adding the second agent to the dialog session, the adding including;

relaying to the second agent, via an API endpoint associated with the second agent, messages sent to and from the first agent and the user device, respectively; and

relaying messages from the second agent via the API endpoint associated with the at least second agent to first agent and the user device over the communication channel.

9. The system of claim 2, wherein creating the dialog session includes at least one of:

receiving a message addressed to the first agent from the user device over the communication channel, the message including contact information of the first agent corresponding to the communication channel;

receiving a request from the user device to add the first agent to a dialog on the communication channel;

or receiving a request to link the first agent to the communication channel.

10. The system of claim 1, the operations further comprising:

adding information corresponding to a communication channel to a directory of communication channels, the information for linking agents or user profiles to the communication channel;

receiving and storing information pertaining to agents in the directory of the plurality of agents;

receiving one or more of a client id or a client token for web services associated with an agent, at least one of the client id or client token allowing the system to;

relay messages on behalf of the agent; and

call services provided by the agent; and

receiving one or more of a client id or a client token associated with authentication of an agent on a communication channel, at least one of the client id or client token allowing the system to relay messages on behalf of the agent over the communication channel.

11. A non-transitory computer-readable medium having stored thereon instructions that, when executed, program one or more processors to perform operations comprising:

registering an agent, the agent including instructions that, when executed, program one or more processors to perform operations comprising;

linking one or more agent communication channels to the agent, wherein linking the one or more agent communication channels to the agent comprises causing an intercom to use data stored in an agent directory to make requests to send or receive messages or files over the one or more agent communication channels on behalf of the agent;

providing dialog-based services to provide information in a user interface via an exchange of natural language, the registering the agent including receiving an application programming interface (API) endpoint associated with the agent and authentication credentials corresponding to multiple communication channels on which the agent has been authenticated for communication to one or more user devices;

receiving requests to dialog with the agent from the multiple communication channels, each request comprising a service trigger, the service trigger being in response to identifying relevant services specified in a natural language communication received from a user of the user device;

based at least in part on the service trigger, creating a dialog session between the agent and the multiple communication channels using the one or more agent communication channels;

capturing dialog between the agent and the user device via the dialog session;

detecting a change of activity on a first communication channel;

detecting a change of activity on a second communication channel; and

maintaining the dialog session between the agent and the multiple communication channels using the API endpoint and the authentication credentials of the agent for communicating with the one or more user devices via the second communication channel, the second communication channel being one of the one or more agent communication channels.

12. The non-transitory computer-readable medium of claim 11, wherein relaying dialog between the agent and one of the multiple communication channels includes:

receiving messages from one of the multiple communication channels on behalf of the agent using a sub-group of the authentication credentials;

relaying the messages to the agent via the API endpoint;

receiving responses from the agent via the API endpoint; and

relaying the responses to one of the multiple communication channels on behalf of the agent using the authentication credentials.

13. The non-transitory computer-readable medium of claim 12 wherein using the authentication credentials includes authenticating a central service to act on behalf of the agent, the central service configured for:

registering the agent, receiving requests to dialog with the agent, receiving the messages, relaying the messages, receiving the responses, and relaying the responses.

14. The non-transitory computer-readable medium of claim 13, the operations further comprising:

storing a user profile having one or more user-communication channels associated therewith; and

tracking a dialog session between the agent and a user-communication channel associated with the user profile, the tracking including associating a dialog identifier with the agent, the user-communication channel, and the user profile.

15. The non-transitory computer-readable medium of claim 14, the operations further comprising:

receiving a message from a different communication channel associated with the user profile; and

continuing the dialog between the agent and a user device by using the different communication channel, the continuing including additionally associating the different communication channel with the dialog identifier or alternatively associating the different communication channel with the dialog identifier.

16. A method comprising:

intercepting communication in a communication channel from a plurality of communication channels;

identifying a service trigger from the intercepted communication, the service trigger being in response to identifying relevant services specified in a natural language communication received from a user of the user device;

identifying a first agent from the plurality of agents associated with a service corresponding to the service trigger, each agent being a computer program that, when executed, implements a dialog-based service to provide information in a user interface via an exchange of natural language, the plurality of communication channels being shared by the plurality of agents, each agent being linkable to one or more of the communication channels;

linking one or more first-agent communication channels from the plurality of communication channels to the first agent, wherein linking the one or more first-agent communication channels to the first agent comprises causing an intercom to use data stored in an agent directory to make requests to send or receive messages or files over the one or more first-agent communication channels on behalf of the first agent;

relaying dialog between the first agent and a first communication channel endpoint associated with a user device for presenting the user interface;

identifying a change from the first communication channel endpoint to an endpoint associated with a second communication channel;

based at least in part on the change in the communication channel, maintaining a dialog session between the first agent and the user device using the endpoint associated with the second communication channel, wherein maintaining the dialog session includes continuing the dialog between the first agent and the user device and routing messages via an API endpoint associated with the first agent over the second communication channel, the second communication channel being one of the first-agent communication channels.

17. The method of claim 16, wherein the service trigger includes one or more of:

keywords that are associated with the service, subject matter corresponding to the service, a received user location corresponding to the service, a time corresponding to the service, a data type corresponding to the service, a request to add an agent to a conversation, a reference to an agent, or a request for a service.

18. The method of claim 16 further comprising:

detecting one or more of a fidelity of the communication channel or constraints of the communication channel; and

conforming responses of the first agent to one or more of the fidelity of the communication channel or the constraints of the communication channel.