Virtual assistant system to enable actionable messaging

• US 9,661,105 B2
• Filed: 12/11/2015
• Issued: 05/23/2017
• Est. Priority Date: 12/11/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Wand Labs, Inc. (Microsoft Corporation)
• Inventors: Sharma, Vishal, Amjadi, Elhum
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US14/966,212
• Publication Number:
  US 20160173578A1
• Time in Patent Office: 529 Days
• Field of Search: 701300, 345582, 345619, 345173, 345419, 345473, 345587
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/18 Commands or executable codes
• CPC Class Codes: H04L 63/0884 by delegation of authentication, e.g. a proxy authenticates an entity to be authenticated on behalf of this entity vis-à-vis an authentication entity
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04M 1/72409 by interfacing with external accessories hands-free H04M1/60
• CPC Class Codes: H04M 1/724094 Interfacing with a device worn on the user's body to provide access to telephonic functionalities, e.g. accepting a call, reading or composing a message
• CPC Class Codes: H04M 1/724097 Worn on the head
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/43 : using shared identity modules, e.g. SIM sharing
• CPC Class Codes: H04W 4/00 : Services specially adapted for wireless communication networks; Facilities therefor
• CPC Class Codes: H04W 88/02 : Terminal devices

Abstract

A virtual assistant system includes a mobile device to receive an input command corresponding to a function to be performed at one or more external services, to translate the input command into a semantic atom representing the command, and to transmit the semantic atom, and an assistant server configured to receive the transmitted semantic atom, the assistant server including a plurality of plugins, each plugin corresponding to a respective one of the external services and configured to generate a proprietary language command corresponding to the received semantic atom for each external service in which the function is to be performed.

Claims

1. A virtual assistant system, comprising

an assistant server configured to:

receive from a mobile device a first semantic atom corresponding to an input command for one or more external services, the assistant server configured to generate a language command specific to each of the one or more external services corresponding to the first semantic atom; and

transmit a second semantic atom comprising the generated language command back to the mobile device to enable the mobile device to directly control the one or more external services using a wireless connection.

2. The virtual assistant system of claim 1, wherein:

the assistant server is further configured to receive output data from the one or more external services at one or more plugins executed by the assistant server as a result of performing a function specified by the received semantic atom, and to add the received output data to the second semantic atom.

3. The virtual assistant system of claim 1, wherein the assistant server is configured to combine a plurality of semantic atoms to create a single merged semantic atom.

4. The virtual assistant system of claim 1, wherein the assistant server is configured to separate the first semantic atom into a plurality of semantic atoms, each of the plurality of semantic atoms including a subset of information contained in the first semantic atom.

5. The virtual assistant system of claim 1, wherein at least one of the mobile device and the assistant server is configured to transmit one or more semantic atoms to at least one other mobile device.

6. The virtual assistant system of claim 1, wherein the assistant server is further configured to delete the first semantic atom after generating the language command when the first semantic atom is determined to no longer be required for further information transmissions thereof with other services or the mobile device.

7. The virtual assistant system of claim 1, wherein the assistant server is further configured to store the first semantic atom after generating the proprietary language command.

8. The virtual assistant system of claim 1, wherein:

the assistant server provides the first semantic atom to a plurality of plugins to generate language commands for each of the one or more external services when the input command corresponds to a function or functions to be performed at the one or more external services.

9. The virtual assistant system of claim 1, wherein the assistant server stores credentials including data to authorize the assistant server to transmit the generated language command to the one or more external services.

10. The virtual assistant system of claim 9, wherein the first semantic atom extends authority to control the one or more external services to at least one other mobile device.

11. The virtual assistant system of claim 10, wherein a further semantic atom received from the mobile device revokes the authorization of the at least one other mobile device to control the one or more external services.

12. The virtual assistant system of claim 10, wherein, in response to receiving the first semantic atom to extend authority, the assistant server is configured to copy the credentials of the mobile device to a separate assistant server accessed by the at least one other mobile device.

13. The virtual assistant system of claim 1, wherein the assistant server analyzes context information of the input command to modify the semantic atom.

14. The virtual assistant system of claim 13, wherein the context information analyzed by the assistant server includes at least one of a location of the mobile device when the command is input, a time of day when the command is input, co-presence of specific individuals, involuntary actions by a user, and action sequences taken by a user of the mobile device.

15. The virtual assistant system of claim 14, wherein the action sequences taken by the user include a plurality of input commands made simultaneously or sequentially.

16. The virtual assistant system of claim 14, wherein the action sequences taken by the user include an input command that is invalidated and followed by another input command.

17. The virtual assistant system of claim 1, wherein the assistant server comprises a plurality of server devices, the plurality of server devices being configured to transfer semantic atoms amongst each other.

18. The virtual assistant system of claim 17, wherein the plurality of server devices comprises:

a main server device configured to communicate with the mobile device; and

one or more subordinate server devices configured to communicate with the main server device.

19. The virtual assistant system of claim 17, wherein:

one or more of the server devices stores one or more plugins, each of the one or more plugins corresponding to a different one of the one or more external services; and

each of the server devices, upon receiving a semantic atom representing an input command corresponding to a function to be performed at an external service, transmits the semantic atom to the server device storing the plugin corresponding to the external service.

20. A virtual assistant system, comprising:

a mobile device to receive an input command corresponding to a function to be performed at one or more external services, to translate the input command into a semantic atom representing the command, and to transmit the semantic atom; and

an assistant server configured to receive the transmitted semantic atom, the assistant server including a plurality of plugins, each plugin corresponding to a respective one of the external services and configured to generate a proprietary language command corresponding to the received semantic atom for each external service in which the function is to be performed;

wherein the assistant server stores credentials including data to authorize the mobile device to perform input commands for the one or more external services;

wherein an input command at the mobile device extends authority to perform input commands for the one or more external services to at least one other mobile device;

wherein the input command to extend authorization to the at least one other mobile device to perform input commands to the one or more external services includes information to limit the authorization to a predetermined level of control, a predetermined location in which control is permitted, a predetermined time frame in which the control is permitted and a predetermined access path in which another user approaches the one or more external services.

21. A method of controlling external services, the method comprising:

receiving a first semantic atom from a mobile device, the first semantic atom comprising an input command corresponding to a function to be performed at a selected one of one or more external services;

generating a language command specific to each of the one or more external services corresponding to the first semantic atom;

transmitting a second semantic atom comprising the generated language command back to the mobile device to enable the mobile device to directly control the one or more external services using a wireless connection.

22. A server device, comprising:

a storage system configured to store semantic atoms which can be shared between a plurality of external services, the storage system further including a library of commands to perform functions at the plurality of external services;

a plurality of plugins, each plugin corresponding to a respective one of the plurality of external services, each plugin being configured to translate between semantic atoms and a respective proprietary language of the corresponding external service; and

a platform configured to share semantic atoms between the plurality of external services by using the proprietary language translated at the respective plugins and to share semantic atoms with a mobile device in which commands to perform a function are initially input and transmitted to the platform as sematic atoms;

the server device configured to generate a second semantic atom comprising a generated language command in response to receiving a semantic atom from the mobile device, and to transmit the second semantic atom back to the mobile device to enable the mobile device to directly control the one or more external services using a wireless connection.

23. The server device of claim 22, wherein the storage system stores at least one command set corresponding to a respective one of the plurality of external services, each command set including commands to perform corresponding functions at the respective external service.

24. The server device of claim 22, wherein each plugin is created at the server device to include a set of commands selected from the stored library of commands, the set of commands to correspond with associated functions to be performed at the corresponding service.

25. The server device of claim 22, wherein the platform is configured to wirelessly connect to a plurality of mobile devices to share information therebetween and to receive input commands therefrom, the information being shared and input commands being received being provided as semantic atoms.

26. A mobile device, comprising:

a sensor configured to receive an input command to perform a function at one or more external services; and

an assistant interface configured to translate the received input command into a first semantic atom and to send the first semantic atom to an external server to perform the function at the one or more external services via a corresponding plugin and to receive a second semantic atom including language command of the one or more external services to directly control the one or more external services using a wireless connection.