Virtual assistant system to enable actionable messaging

• US 9,935,904 B2
• Filed: 06/03/2016
• Issued: 04/03/2018
• Est. Priority Date: 12/11/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sharma, Vishal, Amjadi, Elhum
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/173,053
• Publication Number:
  US 20170230316A1
• Time in Patent Office: 669 Days
• Field of Search: 701300, 345582, 345619, 345173, 345419, 345473, 345587
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/18 Commands or executable codes
• CPC Class Codes: H04L 63/0884 by delegation of authentication, e.g. a proxy authenticates an entity to be authenticated on behalf of this entity vis-à-vis an authentication entity
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04M 1/72409 by interfacing with external accessories hands-free H04M1/60
• CPC Class Codes: H04M 1/724094 Interfacing with a device worn on the user's body to provide access to telephonic functionalities, e.g. accepting a call, reading or composing a message
• CPC Class Codes: H04M 1/724097 Worn on the head
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/43 : using shared identity modules, e.g. SIM sharing
• CPC Class Codes: H04W 4/00 : Services specially adapted for wireless communication networks; Facilities therefor
• CPC Class Codes: H04W 88/02 : Terminal devices

Abstract

A virtual assistant system to control external services via messaging includes a mobile device configured to participate in a messaging session between a plurality of other participants, to receive an input command from a user, to translate the received input command into a semantic atom, and to transmit the semantic atom via a message to the messaging session, and an assistant server configured to receive the semantic atom via the message, the assistant server including at least one plugin, the plugin corresponding to a respective external service and configured to generate a proprietary language command corresponding to the received semantic atom, the proprietary language command controlling the corresponding external service to perform a function corresponding to the input command.

Claims

1. A virtual assistant system to control external services via messaging, the virtual assistant system comprising:

a mobile device configured to participate in a messaging session between a plurality of other participants, to receive an input command from a user, to translate the received input command into a semantic atom, and to transmit the semantic atom via an actionable message to the messaging session, the messaging session supporting a plurality of messaging functions including receiving, transmitting, and displaying messages targeted to one or more of the plurality of participants; and

an assistant server configured to receive the semantic atom via the actionable message, the assistant server including at least one plugin, the plugin corresponding to a respective external service and configured to generate a proprietary language command corresponding to the received semantic atom, the proprietary language command controlling the corresponding external service to perform an external function corresponding to the input command;

wherein the external function performed by each external service comprises a function other than the messaging functions supported by the messaging session.

2. The virtual assistant system of claim 1, wherein the at least one plugin is further configured to receive output data from the corresponding external service as a result of performing the function, to convert the received output data to one or more semantic atoms representing the received output data, and to send the one or more semantic atoms representing the received output data via messages to the messaging session.

3. The virtual assistant system of claim 2, wherein:

at least one of the plurality of participants has authority to issue commands to the at least one plugin; and

the at least one plugin sends the one or more semantic atoms representing the received output data to the at least one of the plurality of participants having authority to issue commands to the at least one plugin.

4. The virtual assistant system of claim 1, wherein each of the plurality of participants in the messaging session receives the semantic atom via the message that is sent to the messaging session.

5. The virtual assistant system of claim 1, wherein the plurality of participants includes at least one of the assistant server, another mobile device, and an external service.

6. The virtual assistant system of claim 5, wherein an external service participates in the messaging session via a corresponding plugin.

7. The virtual assistant system of claim 1, wherein the mobile device includes an assistant interface configured to receive the input command from the user and to translate the input command into the semantic atom.

8. The virtual assistant system of claim 1, wherein at least one of the plurality of participants invites an external service to become a participant of the messaging session.

9. The virtual assistant system of claim 1, wherein the assistant server stores credentials including data to authorize one or more participants of the messaging session to issue commands for the corresponding external service.

10. The virtual assistant system of claim 9, wherein:

at least one of the participants of the messaging session is an external service; and

the plugin corresponding to the external service participating in the messaging session transmits one or more semantic atoms via messages only to participants of the messaging session having authority to issue commands to the external service.

11. The virtual assistant system of claim 9, wherein:

at least one participant of the messaging session controls the assistant server to extend authority to input commands to the corresponding external service to at least one other participant of the messaging session.

12. The virtual assistant system of claim 9, wherein:

a plurality of the participants of the messaging session control the assistant server to extend authority to input commands to the corresponding external service to each other.

13. The virtual assistant system of claim 1, wherein a participant of the messaging session transmits the semantic atom representing the input command via a message to another messaging session between another plurality of participants.

14. The virtual assistant system of claim 1, wherein:

at least one other participant of the messaging session adds data to the semantic atom.

15. The virtual assistant system of claim 1, wherein the assistant server analyzes context information of the semantic atom to determine the function to be performed before generating the proprietary language command.

16. The virtual assistant system of claim 15, wherein the context information analyzed by the assistant server includes at least one of a number of times the semantic atom has been transmitted between participants of the messaging session and content of messages transmitted in the messaging session.

17. The virtual assistant system of claim 16, wherein the assistant server extends authority to issue commands to at least one external service to one or more participants of the messaging session based on the content of messages transmitted in the messaging session.

18. The virtual assistant system of claim 17, wherein the content of the messages transmitted in the messaging session includes a request for authority to issue commands and a response to the request.

19. The virtual assistant system of claim 18, wherein the response to the request is automated.

20. The virtual assistant system of claim 16, wherein the assistant server extends authority to make transactions using payment credentials of a participant based on the content of messages transmitted in the messaging session.

21. The virtual assistant system of claim 1, wherein the assistant server provides the received atom to a plurality of plugins to generate a corresponding plurality of proprietary language commands.

22. The virtual assistant system of claim 1, wherein the assistant server provides the received semantic atom to a plurality of plugins to perform a corresponding plurality of functions.

23. The virtual assistant system of claim 1, wherein the assistant server comprises a plurality of server devices, the plurality of server devices being configured to transmit semantic atoms between each other via a corresponding message via the messaging session.

24. The virtual assistant system of claim 23, wherein the plurality of server devices comprises:

a main server device configured to communicate with the mobile device; and

one or more subordinate server devices configured to communicate with the main server device.

25. The virtual assistant system of claim 23, wherein:

one or more of the server devices stores one or more plugins, each of the one or more plugins corresponding to a different external service; and

each of the server devices, upon receiving a semantic atom representing an input command corresponding to a function to be performed at an external service, transmits the semantic atom via a message to the server device storing the plugin corresponding to the external service.

26. The virtual assistant system of claim 23, wherein one or more of the plurality of server devices are participants in the messaging session.

27. The virtual assistant system of claim 23, wherein:

one of the plurality of server devices stores credentials including data to authorize one or more participants of the messaging session to issue commands for the one or more external services; and

the server device storing the credentials sends a semantic atom including the stored credentials to another of the plurality of server devices.

28. The virtual assistant system of claim 1, wherein the assistant server is a participant in the messaging session.

29. The virtual assistant system of claim 1, wherein the messaging session is a virtual reality (VR) session.

30. A server device to communicate via messaging, the server device comprising:

a storage system to store semantic atoms which can be shared between a plurality of external services and mobile devices, the storage system further including a library of commands to perform external functions at the plurality of external services;

a plurality of plugins, each plugin corresponding to a respective one of the plurality of external services, each plugin being configured to translate between semantic atoms and a respective proprietary language of the corresponding external service; and

a platform configured to share semantic atoms between the plurality of external services and mobile devices via messaging, the platform sharing semantic atoms with the plurality of external services using the proprietary language translated at the respective plugins and sharing semantic atoms with the plurality of mobile devices in which commands to perform any of the external functions are initially input and transmitted to the platform as sematic atoms, each of the external functions comprising a function other than messaging functions associated with the messaging.

31. The server device of claim 30, wherein the server device participates in a messaging session between a plurality of other participants, the plurality of participants including at least one of an external service and a mobile device.

32. The server device of claim 30, wherein the platform is configured to directly control external services via messaging by providing semantic atoms to respective plugins corresponding to the external services to generate proprietary language commands.

33. The server device of claim 30, wherein the server device adds data to a semantic atom and transmits the semantic atom to an external service or mobile device via a message.

34. The server device of claim 30, further comprising a plurality of server devices, the plurality of server devices being configured to transmit semantic atoms between each other via corresponding messages.

35. The virtual assistant system of claim 34, wherein the plurality of server devices comprises:

a main server device configured to communicate with the mobile device; and

one or more subordinate server devices configured to communicate with the main server device.

36. The virtual assistant system of claim 34, wherein:

one or more of the server devices stores one or more plugins, each of the one or more plugins corresponding to a different external service; and

each of the server devices, upon receiving a semantic atom representing an input command corresponding to a function to be performed at an external service, transmits the semantic atom via a message to the server device storing the plugin corresponding to the external service.

37. The virtual assistant system of claim 34, wherein:

one of the plurality of server devices stores credentials including data to authorize one or more participants of a messaging session to issue commands for the one or more external services; and

the server device storing the credentials sends a semantic atom including the stored credentials to another of the plurality of server devices.

38. A mobile device, comprising:

a user interface to receive an input command from a user; and

an assistant interface to translate the received input command into a semantic atom representing the message, and to transmit the semantic atom via a message to a messaging session to be received at an external server to perform a function corresponding to the input command via a corresponding plugin or to receive a semantic atom including proprietary language of one or more external services to directly perform the function at the one or more external services.

39. The mobile device of claim 38, wherein the assistant interface is configured to transmit the semantic atom to at least one of another mobile device and a server device via a message to a messaging session.

40. The mobile device of claim 38, wherein the assistant interface is configured to directly perform the function at a selected external service by transmitting a semantic atom via a message to the selected external service.