Unified virtual reality platform

• US 10,353,474 B2
• Filed: 09/28/2016
• Issued: 07/16/2019
• Est. Priority Date: 09/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sharma, Vishal, Amjadi, Elhum
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/279,215
• Publication Number:
  US 20170090589A1
• Time in Patent Office: 1,021 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06Q 50/16 Real estate
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

Techniques for providing a unified computational platform for implementing virtual reality systems. In an aspect, a virtual reality (VR) client includes input and output modules for generating a VR user experience. The client also incorporates an assistant interface that couples the input and output modules to an assistant server using a semantic information interchange language (SIIL). The assistant server includes a command inference block for converting user-input SIIL messages to command-input SIIL messages, thereby encoding user VR inputs into a general command set that is recognized and processed by both VR and non-VR modules. Further aspects include techniques for seamless integration between virtual environments and joint VR sessions supporting multiple users.

Claims

1. A server apparatus comprising a central processing unit (CPU), and a storage unit storing instructions executable by the CPU, the instructions configured to implement:

a command inference block configured to, using a communicator, receive from a client a user-input message according to a universal protocol specifying an input to a virtual reality system, and to convert the user-input message into a command-input message according to the universal protocol; and

at least one plugin module configured to;

transmit using a communicator a command input specified by the command-input message to an application module;

translate a custom service message received from the application module into a universal service message according to the universal protocol; and

transmit using a communicator the universal service message to the client.

2. The apparatus of claim 1, the user-input message specifying at least one of a gesture performed by the user and a voice input.

3. The apparatus of claim 1, the command inference block configured to convert the user-input message into the command-input language by being configured to:

receive a context of the user-input message;

retrieve a proposed command from a library based on the received context and the user-input message; and

encapsulate the proposed command in the command-input message.

4. The apparatus of claim 1, the context comprising at least one of co-presence of other individuals in a joint VR environment when one user generates an input, the user'"'"'s geographic location when generating the input, a time of day when the user input is made, and a direction in which a physical input such as a gesture is made.

5. The apparatus of claim 1, the command inference block configured to convert the user-input message into the command-input language by being configured to:

receive a context of the user-input message;

retrieve a proposed command from a library based on the received context and the user-input message;

if a rejection of the proposed command is received from the user, encapsulate another command suggested by the user in the command-input message; and

update the library based on the another command suggested by the user.

6. The apparatus of claim 1, the command inference block configured to convert the user-input message into the command-input language by being configured to:

receive a context of the user-input message;

retrieve a proposed command from a library based on the received context and the user-input message;

if the library does not contain a proposed command based on the received context and the user-input message, encapsulate another command suggested by a second user in the command-input message; and

update the library based on the another command suggested by the second user.

7. The apparatus of claim 1, the command inference block configured to convert the user-input message into the command-input language by being configured to:

receive a plurality of user-input messages from a plurality of users;

receive a context of each user-input message;

retrieve a proposed command from a library based on the plurality of received contexts and the plurality of user-input messages; and

encapsulate the proposed command in the command-input message.

8. The apparatus of claim 1, the at least one plugin module comprising a first virtual reality (VR) plugin and a second non-virtual reality (non-VR) plugin, the first VR plugin configured to exchange at least one parameter of the user with the second non-VR plugin using a message according to the universal protocol, the at least one parameter comprising a credential or authorization associated with a payment instrument owned by the user, an assigned status of the user, and a user profile.

9. The apparatus of claim 1, the at least one plugin module comprising a first virtual reality (VR) plugin and a second virtual reality (VR) plugin, the first VR plugin configured to exchange with the second VR plugin using messages according to the universal protocol a virtual environment parameter comprising at least one of a virtual environment coordinate system, relative position of a user in the virtual environment coordinate system, and visual data associated with the virtual environment.

10. The apparatus of claim 9, the first VR plugin configured to exchange data with the second VR plugin using a network communications channel.

11. The apparatus of claim 1, the universal service message comprising an instruction to the client to change an output signal generated by a virtual reality output module.

12. The apparatus of claim 1, further comprising a server computer for implementing the command inference block and the at least one plugin module, the server computer configured to communicate with the client using a network connection.

13. The apparatus of claim 1, the command inference block further configured to receive a second user-input message according to the universal protocol, and to convert the combination of the user-input message and the second user-input message into a single command-input message according to the universal protocol.

14. A method for converting a user-input digital message to a command-input digital message for a virtual reality system comprising a communicator, a central processing unit (CPU), and a storage unit storing instructions executable by the CPU, the method comprising:

receiving using the communicator at least one user-input message according to a universal protocol;

receiving a context for each of the at least one user-input message;

retrieving a proposed command, if available, from a digital library mapping user input and context to command inputs;

if a user rejection of the proposed command is not received, encapsulating the proposed command in at least one command-input message according to the universal protocol; and

transmitting using the communicator the command-input message to an application module.

15. The method of claim 14, further comprising:

if a user rejection of the proposed command is received, encapsulating an alternative command specified by a user in the at least one command-input message according to the universal protocol; and

updating the digital library to map the user input and context to the alternative command specified by the user.

16. The method of claim 14, further comprising:

if the proposed command is not available from the digital library mapping user input and context to command inputs, receiving an alternative command specified by an alternative user; and

encapsulating the alternative command in the at least one command-input message according to the universal protocol.

17. The method of claim 16, the receiving at least one user-input message according to a universal protocol comprising receiving at least one user-input message from each of a plurality of users, the retrieving the proposed command comprising retrieving a single proposed command for the plurality of user-input messages from the plurality of users.

18. A computing device including a communicator, a processor, and a memory holding instructions executable by the processor to:

receive from a client using the communicator a user-input message according to a universal protocol specifying an input to a virtual reality system;

convert the user-input message into a command-input message according to the universal protocol;

transmit using the communicator a command input specified by the command-input message to an application module;

translate a custom service message received from the application module into a universal service message according to the universal protocol; and

transmit using the communicator the universal service message to the client.

19. The device of claim 18, the user-input message specifying at least one of a gesture performed by the user and a voice input.

20. The device of claim 18, the memory further holding instructions executable by the processor to:

receive a context of the user-input message;

retrieve a proposed command from a library based on the received context and the user-input message; and

encapsulate the proposed command in the command-input message.