Method and system for improving the transmission and processing of data regarding a multi-user virtual environment

• US 10,726,625 B2
• Filed: 01/28/2015
• Issued: 07/28/2020
• Est. Priority Date: 01/28/2015
• Status: Active Grant

• Current Assignee: CCP hf (PearlAbyss Corp.)
• Original Assignee: CCP hf (PearlAbyss Corp.)
• Inventors: Kraver, Adam
• Primary Examiner(s): Wu, Ming
• Application Number:
  US14/608,054
• Publication Number:
  US 20160217615A1
• Time in Patent Office: 2,008 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092

Abstract

The present disclosure provides computer systems, apparatuses, computer-executable methods and one or more non-transitory computer-readable media for implementing a multi-user virtual reality environment. An example of a computer-implemented method for implementing a multi-user virtual reality environments includes receiving sensor data from a sensor at a local device, providing the sensor data to a network interface for electronic transmission to a remote device, providing the sensor data to a virtual environment engine executing on a processor of the local device, determining, by the virtual environment engine and based at least in part on the sensor data, virtual environment data related to a virtual reality environment, providing the virtual environment data to the network interface for electronic transmission to the remote device, and electronically transmitting, using the network interface, the sensor data and the virtual environment data to the remote device for use in generating a multi-user virtual environment.

Claims

1. A computer-implemented method for implementing a multi-user virtual reality environment, the method comprising:

receiving sensor data from a sensor at a local device;

providing the sensor data to a network interface for electronic transmission to a remote device;

electronically transmitting, using the network interface and via a first set of packets explicitly reserved for transmission of only the sensor data, the sensor data to the remote device for use in generating a multi-user virtual environment, wherein electronically transmitting the sensor data via the first set of packets explicitly reserved for transmission of the sensor data enables faster updating of a display of the remote device;

providing the sensor data to a virtual environment engine executing on a processor of the local device;

determining, by the virtual environment engine and based at least in part on the sensor data, virtual environment data related to a virtual reality environment, the virtual environment data being separate from and exclusive of the sensor data;

separately from providing the sensor data to the network interface, providing the virtual environment data to the network interface for electronic transmission to the remote device;

electronically transmitting, using the network interface and via a second set of packets explicitly reserved for transmission of virtual environment data, the virtual environment data to the remote device for use in generating the multi-user virtual environment, wherein electronically transmitting the virtual environment data via the second set of packets explicitly reserved for transmission of the virtual environment data further enables faster updating of the display of the remote device; and

providing the sensor data to another local device to verify a legitimacy of any actions indicated in the virtual environment data,

wherein;

the sensor data is transmitted at a first interval and the virtual environment data is transmitted at a second interval, the second interval being different than the first interval,

the sensor data and the virtual environment data are separately transmitted via different sockets,

the sensor data comprises at least one frame of data received from a sensor, and

the sensor data further comprises a set of mask data identifying one or more pixels within the at least one frame of data that are associated with at least one object.

2. The method of claim 1, wherein the network interface comprises at least two separate network connections to the remote device, and wherein each of the sensor data and the virtual environment data is transmitted via a different connection of the at least two separate network connections.

3. The method of claim 1, further comprising post-processing the sensor data prior to transmission to the remote device.

4. The method of claim 3, wherein post-processing comprises at least one of filtering the sensor data, smoothing the sensor data, cropping the sensor data, or identifying the presence of at least one object within the sensor data.

5. The method of claim 1, wherein the sensor data further comprises cached background data identifying the presence of at least one object not visible within the at least one frame of data.

6. A computer-implemented method for implementing a multi-user virtual reality environment, the method comprising:

receiving, at a local sensor interface and via a first set of packets explicitly reserved for transmission of sensor data, only a first set of sensor data from a remote device via a network interface, the first set of sensor data being received at a first interval;

receiving a second set of sensor data from a local sensor by the local sensor interface;

providing, by the local sensor interface, the first set of sensor data and the second set of sensor data to a local virtual environment engine executing on a processor;

receiving, by the local virtual environment engine and via a second set of packets explicitly reserved for transmission of virtual environment data, a set of virtual environment data from the remote device via the network interface, wherein the first set of sensor data and the set of virtual environment data are received via different sockets, and wherein the set of virtual environment data is separate from and exclusive of the first and second sets of sensor data;

generating a multi-user virtual environment based in least in part on both the first set of sensor data and the second set of sensor data;

providing instructions to a display device to display at least a portion of the multi-user virtual environment, wherein receipt of the first set of sensor data via the first set of packets explicitly reserved for transmission of only the sensor data at a first interval and receipt of the set of virtual environment data via the second set of packets explicitly reserved for transmission of the virtual environment data at a second interval different than the first interval enables faster updating of the display device;

verifying the set of virtual environment data based at least in part using the first set of sensor data; and

reconstructing an appearance of a user from another remote device based at least in part using portions of the first and the second sets of sensor data that capture at least a position, size, and shape of the user.

7. The method of claim 6, further comprising:

using the first set of sensor data to generate a virtual representation of a user of the remote device; and

presenting the virtual representation of the user of the remote device within the multi-user virtual environment.

8. The method of claim 6, wherein the first set of sensor data further comprises object mask data, and the method comprises determining a presence of a use of the remote device within the first set of sensor data based at least in part on the object mask data.

9. The method of claim 6, further comprising applying at least one post-processing operation to the second set of sensor data, but not the first set of sensor data.

10. The method of claim 9, wherein the first set of sensor data comprises at least one parameter indicating the first set of sensor data was post-processed prior to receipt by the local sensor interface.

11. The method of claim 6, wherein the display device is a head-mounted display.

12. A system for implementing a multi-user virtual reality environment, the system comprising:

sensor interface circuitry configured to;

receive sensor data from a sensor at a local device;

provide the sensor data to a network interface for electronic transmission to a remote device; and

provide the sensor data to a virtual environment engine executing on a processor of the local device;

virtual environment state management circuitry configured to;

determine, based at least in part on the sensor data, virtual environment data related to a virtual reality environment, the virtual environment data being separate from and exclusive of the sensor data; and

separately from providing the sensor data to the network interface, provide the virtual environment data to the network interface for electronic transmission to the remote device; and

the network interface, which is configured to;

electronically transmit, via a first set of packets explicitly reserved for transmission of only the sensor data, the sensor data to the remote device for use in generating a multi-user virtual environment, wherein electronic transmission the sensor data via the first set of packets explicitly reserved for transmission of the sensor data enables faster updating of a display of the remote device;

electronically transmit, via a second set of packets explicitly reserved for transmission of virtual environment data, the virtual environment data to the remote device for use in generating the multi-user virtual environment, wherein electronic transmission of the virtual environment data via the second set of packets explicitly reserved for transmission of the virtual environment data further enables faster updating of the display of the remote device; and

provide the sensor data to another local device to verify a legitimacy of any actions indicated in the virtual environment data,

wherein;

the sensor data is transmitted at a first interval and the virtual environment data is transmitted at a second interval, the second interval being different than the first interval,

the network interface is configured to transmit the sensor data and the virtual environment data via different sockets,

the sensor data comprises at least one frame of data received from a sensor, and

the sensor data further comprises a set of mask data identifying one or more pixels within the at least one frame of data that are associated with at least one object.

13. The system of claim 12, wherein the network interface comprises at least two separate network connections to the remote device, and wherein each of the sensor data and the virtual environment data is transmitted via a different connection of the at least two separate network connections.

14. The system of claim 12, wherein the sensor interface circuitry is further configured to post-process the sensor data prior to transmission to the remote device.

15. The system of claim 12, wherein the sensor data further comprises cached background data identifying the presence of at least one object not visible within the at least one frame of data.

16. The system of claim 12, wherein the sensor data further comprises sensor configuration data indicating at least one configuration parameter of the sensor.

17. A system for implementing a multi-user virtual reality environment, the system comprising:

sensor interface circuitry configured to;

receive a first set of sensor data from a remote device via a network interface and via a first set of packets explicitly reserved for transmission of sensor data the first set of sensor data being received at a first interval;

receive a second set of sensor data from a local sensor; and

provide the first set of sensor data and the second set of sensor data to a local virtual environment engine executing on a processor; and

virtual environment state management circuitry configured to;

generate a multi-user virtual environment based in least in part on both the first set of sensor data and the second set of sensor data;

provide instructions to a display device to display at least a portion of the multi-user virtual environment, wherein receipt of the first set of sensor data via the first set of packets explicitly reserved for transmission of only the sensor data enables faster updating of the display device;

receive a set of virtual environment data from the remote device via the network interface and via a second set of packets explicitly reserved for transmission of virtual environment data, wherein receipt of the virtual environment data via the second set of packets explicitly reserved for transmission of the virtual environment data further enables faster updating of the display device, wherein the virtual environment data is separate from and exclusive of the first and second sets of sensor data, wherein the set of virtual environment data is received at a second interval different than the first interval, and wherein the first set of sensor data and the set of virtual environment data are received via different sockets;

verify the set of virtual environment data based at least in part using the first set of sensor data; and

reconstruct an appearance of a user from another remote device based at least in part using portions of the first and the second sets of sensor data that capture at least a position, size, and shape of the user.

18. The system of claim 17, wherein the virtual environment state management circuitry is further configured to:

use the first set of sensor data to generate a virtual representation of a user of the remote device; and

present the virtual representation of the user of the remote device within the multi-user virtual environment.

19. The system of claim 17, wherein the sensor interface circuitry is further configured to apply at least one post-processing operation to the second set of sensor data, but not the first set of sensor data.

20. The system of claim 17, wherein the display device is a head-mounted display.