Virtual assistant system to enable actionable messaging

• US 10,585,685 B2
• Filed: 05/24/2017
• Issued: 03/10/2020
• Est. Priority Date: 12/11/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sharma, Vishal, Amjadi, Elhum
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/603,454
• Publication Number:
  US 20170264715A1
• Time in Patent Office: 1,021 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/18 Commands or executable codes
• CPC Class Codes: H04L 63/0884 by delegation of authentication, e.g. a proxy authenticates an entity to be authenticated on behalf of this entity vis-à-vis an authentication entity
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04M 1/72409 by interfacing with external accessories hands-free H04M1/60
• CPC Class Codes: H04M 1/724094 Interfacing with a device worn on the user's body to provide access to telephonic functionalities, e.g. accepting a call, reading or composing a message
• CPC Class Codes: H04M 1/724097 Worn on the head
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/43 : using shared identity modules, e.g. SIM sharing
• CPC Class Codes: H04W 4/00 : Services specially adapted for wireless communication networks; Facilities therefor
• CPC Class Codes: H04W 88/02 : Terminal devices

Abstract

A virtual assistant system includes a mobile device to receive an input command corresponding to a function to be performed at one or more external services, to translate the input command into a semantic atom representing the command, and to transmit the semantic atom, and an assistant server configured to receive the transmitted semantic atom, the assistant server including a plurality of plugins, each plugin corresponding to a respective one of the external services and configured to generate a proprietary language command corresponding to the received semantic atom for each external service in which the function is to be performed.

Claims

1. An assistant server comprising one or more processors configured to perform the following operations:

receive, from a computing device, a semantic atom representing an input command from a user, the input command corresponding to a function to be performed at one or more external services;

translate the semantic atom representing the input command into one or more language commands that are specific to one or more external services; and

control the one or more external services by transmitting the one or more language commands corresponding to the semantic atom to the one or more external services.

2. The assistant server of claim 1, wherein the one or more external services presents a virtual environment as part of a virtual reality (VR) session amongst a plurality of participants.

3. The assistant server of claim 1, wherein the one or more external services presents a virtual environment as part of an augmented reality (AR) session amongst a plurality of participants.

4. The assistant server of claim 3, further configured to receive a semantic atom representing output data from the AR session, and to present the output data to the user of the computing device.

5. The assistant server of claim 3, wherein at least one of the plurality of participants includes a plugin.

6. The assistant server of claim 3 wherein at least one of the plurality of participants includes another computing device.

7. The assistant server of claim 3, wherein at least one of the plurality of participants includes another user.

8. The assistant server of claim 7, wherein the semantic atom representing at least one of settings, user preferences, user status, user credentials, and command inputs is transmitted to a second plugin participating in the AR session.

9. The assistant server of claim 8, wherein the semantic atom is transmitted to the second plugin automatically when the user inputs a command to the second plugin.

10. The assistant server of claim 3, wherein a participant in the AR session transmits one or more semantic atoms to one or more other participants in the AR session.

11. The assistant server of claim 3, further configured to store credentials including data to authorize one or more participants of the AR session to issue commands for the one or more AR external services.

12. The assistant server of claim 11, further configured to extend authority to input commands to the one or more AR external services to at least one other participant of the AR session.

13. The assistant server of claim 12, wherein the at least one participant of the AR session sets a level of authority the at least one other participant of the AR session has to input commands to the one or more AR external services.

14. The assistant server of claim 11, further configured to automatically authorize one or more participants of the AR session to control the one or more AR external services through input commands according to the stored credentials, the authorization being based on context information.

15. The assistant server of claim 11, wherein a plurality of the participants of the AR session controls the assistant server to extend authority to input commands to the corresponding AR external service to each other.

16. The assistant server of claim 11, wherein:

the stored credentials include at least one payment credential; and

at least one participant of the AR sessions controls the assistant server to extend authority to use the at least one payment credential to at least one other participant of the AR session.

17. The assistant server of claim 1, further configured to analyze context information of the input command before the input command is converted to a proprietary language command to accurately determine the intended function to be performed by the input command.

18. The assistant server of claim 17, wherein the analyzed context information includes at least one of which plugin is receiving the input command, a status of a participant in an AR session, a time of day when the command is input, co-presence of specific individuals, involuntary actions by a user, and action sequences taken by at least one user.

19. The assistant server of claim 18, wherein the action sequences taken by the at least one user includes a plurality of input commands made simultaneously or sequentially.

20. The assistant server of claim 19, further comprising creating a new consolidated input command to represent performing the cluster of input commands simultaneously and storing the consolidated input command in a library of recognizable input commands.

21. The assistant server of claim 18, wherein the action sequences taken by the at least one user includes an input command that is invalidated and followed by another input command.

22. The assistant server of claim 18, wherein the action sequences taken by the at least one user includes an input command that is made by a first user and another input command that is made by a second user.

23. The assistant server of claim 18, further configured to update a stored library of recognizable input commands according to the analyzed context information.

24. The assistant server of claim 18, further configured to update a stored library of recognizable input commands corresponding to a first AR external service according to analyzed context information corresponding to a second AR external service.

25. A method comprising:

receive, from a computing device, a semantic atom representing an input command from a user, the input command corresponding to a function to be performed at one or more external services;

translate the semantic atom representing the input command into one or more language commands that are specific to one or more external services; and

control the one or more external services by transmitting the one or more language commands corresponding to the semantic atom to the one or more external services.

26. The method of claim 25, further comprising presenting a virtual environment as part of a virtual reality (VR) session amongst a plurality of participants.

27. The method of claim 25, further comprising presenting a virtual environment as part of an augmented reality (AR) session amongst a plurality of participants.