Virtual assistant system to enable virtual reality

• US 9,692,855 B2
• Filed: 05/20/2016
• Issued: 06/27/2017
• Est. Priority Date: 12/11/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Wand Labs, Inc. (Microsoft Corporation)
• Inventors: Sharma, Vishal, Amjadi, Elhum
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/160,038
• Publication Number:
  US 20160269508A1
• Time in Patent Office: 403 Days
• Field of Search: 701300, 345582, 345619, 345173, 345419, 345473, 345587
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: H04L 51/00 User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/18 Commands or executable codes
• CPC Class Codes: H04L 63/0884 by delegation of authentication, e.g. a proxy authenticates an entity to be authenticated on behalf of this entity vis-à-vis an authentication entity
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04M 1/72409 by interfacing with external accessories hands-free H04M1/60
• CPC Class Codes: H04M 1/724094 Interfacing with a device worn on the user's body to provide access to telephonic functionalities, e.g. accepting a call, reading or composing a message
• CPC Class Codes: H04M 1/724097 Worn on the head
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/43 : using shared identity modules, e.g. SIM sharing
• CPC Class Codes: H04W 4/00 : Services specially adapted for wireless communication networks; Facilities therefor
• CPC Class Codes: H04W 88/02 : Terminal devices

Abstract

A virtual assistant system to manage a virtual reality (VR) session between a plurality of participants, and includes a mobile device configured to receive an input command from a user participating in the VR session, to translate the input command into a semantic atom representing the command, and to transmit the semantic atom, the input command corresponding to a function to be performed at one or more VR external services, and an assistant server configured to manage the VR session, the assistant server including a plurality of plugins, each plugin corresponding to a respective one of the VR external services and configured to generate a proprietary language command corresponding to the semantic atom, the proprietary language command controlling the corresponding VR external service to perform a function corresponding to the semantic atom.

Claims

1. A virtual assistant system to manage a virtual reality (VR) session between a plurality of participants, the virtual assistant system comprising:

a mobile device configured to receive an input command from a user participating in the VR session, to translate the input command into a semantic atom representing the command, and to transmit the semantic atom, the input command corresponding to a function to be performed at one or more VR external services; and

an assistant server configured to manage the VR session, the assistant server including a plurality of plugins, each plugin corresponding to a respective one of the VR external services and configured to generate a proprietary language command corresponding to the semantic atom, the proprietary language command controlling the corresponding VR external service to perform a function corresponding to the semantic atom.

2. The virtual assistant system of claim 1, wherein the mobile device is further configured to receive a semantic atom representing output data from the VR session, and to present the output data to a user participating in the VR session.

3. The virtual assistant system of claim 1, wherein the mobile device comprises at least one sensor to receive the input command.

4. The virtual assistant system of claim 3, wherein the at least one sensor comprises a stationary sensor.

5. The virtual assistant system of claim 3, wherein the at least one sensor comprises a wearable sensor worn by a user participating in the VR session.

6. The virtual assistant system of claim 1, wherein at least one other participant in the VR session includes at least one of a plugin, another mobile device, another assistant server, and another user.

7. The virtual assistant system of claim 6, wherein a participant in the VR session transmits one or more semantic atoms to one or more other participants in the VR session.

8. The virtual assistant system of claim 7, wherein a first plugin participating in the VR session transmits a semantic atom representing at least one of settings, user preferences, user status, user credentials, and command inputs to a second plugin participating in the VR session.

9. The virtual assistant system of claim 8, wherein the first plugin transmits the semantic atom representing at least one of settings, user preferences, user status, user credentials, and command inputs to the second plugin automatically when the user inputs a command to the second plugin.

10. The virtual assistant system of claim 1, wherein the assistant server analyzes context information of the input command before the input command is converted to a proprietary language command to accurately determine the intended function to be performed by the input command.

11. The virtual assistant system of claim 10, wherein the context information analyzed by the assistant server includes at least one of which plugin is receiving the input command, a status of a user participating in the VR session, a time of day when the command is input, co-presence of specific individuals, involuntary actions by a user, and action sequences taken by at least one user.

12. The virtual assistant system of claim 11, wherein the action sequences taken by the at least one user includes a plurality of input commands made simultaneously or sequentially.

13. The virtual assistant system of claim 11, wherein the action sequences taken by the at least one user includes an input command that is invalidated and followed by another input command.

14. The virtual assistant system of claim 11, wherein the action sequences taken by the at least one user includes an input command that is made by a first user and another input command that is made by a second user.

15. The virtual assistant system of claim 11, wherein the assistant server updates a stored library of recognizable input commands according to the analyzed context information.

16. The virtual assistant system of claim 15, wherein the assistant server updates a stored library of recognizable input commands corresponding to a first VR external service according to analyzed context information corresponding to a second VR external service.

17. The virtual assistant system of claim 15, wherein when the assistant server recognizes a repeated cluster of input commands, the assistant server creates a new consolidated input command to represent performing the cluster of input commands simultaneously and stores the consolidated input command in a library of recognizable input commands.

18. The virtual assistant system of claim 1, wherein the assistant server comprises a plurality of server devices, the plurality of server devices being configured to transfer semantic atoms between each other.

19. The virtual assistant system of claim 18, wherein the plurality of server devices comprises:

a main server device configured to communicate with the mobile device; and

one or more subordinate server devices configured to communicate with the main server device.

20. The virtual assistant system of claim 18, wherein:

one or more of the server devices stores one or more plugins, each of the one or more plugins corresponding to a different VR external service; and

each of the server devices, upon receiving a semantic atom representing an input command corresponding to a function to be performed at an VR external service, transmits the semantic atom to the server device storing the plugin corresponding to the VR external service.

21. The virtual assistant system of claim 18, wherein each of the plurality of server devices corresponds to a different user participating in the VR session.

22. The virtual assistant system of claim 1, wherein the assistant server stores credentials including data to authorize one or more participants of the VR session to issue commands for the one or more VR external services.

23. The virtual assistant system of claim 22, wherein:

at least one participant of the VR session controls the assistant server to extend authority to input commands to the one or more VR external services to at least one other participant of the VR session.

24. The virtual assistant system of claim 23, wherein the at least one participant of the VR session sets a level of authority the at least one other participant of the VR session has to input commands to the one or more VR external services.

25. The virtual assistant system of claim 22, wherein the assistant server automatically authorizes one or more participants of the VR session to control the one or more VR external services through input commands according to the stored credentials, the authorization being based on context information.

26. The virtual assistant system of claim 22, wherein a plurality of the participants of the VR session control the assistant server to extend authority to input commands to the corresponding VR external service to each other.

27. The virtual assistant system of claim 22, wherein:

the stored credentials include at least one payment credential; and

at least one participant of the VR sessions controls the assistant server to extend authority to use the at least one payment credential to at least one other participant of the VR session.

28. The virtual assistant system of claim 1, wherein the VR session comprises augmented reality (AR).

29. A mobile device, comprising:

a user interface configured to present output data to a user participating in a virtual reality (VR) session between a plurality of participants;

at least one sensor to receive a user input corresponding to function to be performed at one or more VR external services;

an assistant interface to translate the input command into a semantic atom representing the command; and

a communicator to transmit the semantic atom.

30. The mobile device of claim 29, wherein at least one of the user interface and the at least one sensor comprises a wearable device worn by the user.

31. The mobile device of claim 29, wherein the at least one sensor comprises a stationary sensor.

32. The mobile device of claim 29, wherein the at least one sensor comprises a wearable sensor worn by a user participating in the VR session.

33. The mobile device of claim 29, wherein the assistant interface is further configured to translate semantic atoms received from other participants of the VR session into outputs displayed on the user interface.

34. The mobile device of claim 29, wherein the assistant interface is visibly represented on the user interface.

35. The mobile device of claim 29, wherein the command input is a touch input on an icon displayed in the user interface.

36. A mobile device, comprising:

a touch display to present output data representing a virtual reality (VR) session between a plurality of participants;

at least one sensor to receive a user input corresponding to a function to be performed at one or more VR external services; and

an assistant interface to translate the input command into a semantic atom representing the command and to transmit the semantic atom.