Method and System for Providing Virtual Display of a Physical Environment

• US 20160217616A1
• Filed: 01/28/2015
• Published: 07/28/2016
• Est. Priority Date: 01/28/2015
• Status: Active Grant

• Current Assignee: CCP hf (PearlAbyss Corp.)
• Original Assignee: CCP hf (PearlAbyss Corp.)
• Inventors: Kraver, Adam
• Granted Patent: US 10,725,297 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

The present disclosure provides computer systems, apparatuses, computer-executable methods and one or more non-transitory computer-readable media for implementing a virtual representation of a physical environment using a virtual reality environment. An example method includes receiving a first set of sensor data from a sensor, identifying at least one physical object within a field of view of the sensor from the first set of sensor data, generating a virtual representation of the at least one physical object based at least in part on the first set of sensor data, generating a virtual environment comprising the virtual representation of the at least one physical object and a virtual representation of at least one virtual object, and displaying the virtual environment via a display device.

Claims

1. A computer-implemented method for implementing a virtual representation of a physical environment using a virtual reality environment, the method comprising:

receiving a first set of sensor data from a sensor;

identifying at least one physical object within a field of view of the sensor from the first set of sensor data;

determining a physical position of the at least one physical object using at least the first set of sensor data;

generating a virtual representation of the at least one physical object based at least in part on the first set of sensor data;

generating a virtual environment comprising the virtual representation of the at least one physical object and a virtual representation of at least one virtual object, wherein the virtual environment comprises a set of coordinates which correspond to a physical area in which the at least one physical object is present, and wherein the physical object is positioned in the virtual environment at a coordinate location that maps to the physical position of the at least one physical object; and

sending instructions to a display device to facilitate display of at least a portion of the virtual environment.

2. The method of claim 1, wherein the display device is a head-mounted display.

3. The method of claim 1, further comprising:

storing the virtual representation of the at least one physical object in a set of cached object data;

receiving a second set of sensor data;

determining that the at least one physical object is not present in the second set of sensor data;

retrieving the virtual representation of the at least one physical object from the set of cached object data; and

sending instructions to a display device to display the retrieved virtual representation of the at least one physical object at a last known position of the at least one physical object.

4. The method of claim 3, further comprising determining that the at least one physical object is at least partially occluded prior to retrieving the virtual representation of the physical object form the set of cached object data.

5. The method of claim 1, wherein the at least one physical object is a user.

6. The method of claim 1, wherein identifying the at least one physical object comprises:

providing the first set of sensor data to a graphics processing unit; and

using at least one shader executing on the graphics processing unit to generate at least one polygon corresponding to he at least one physical object from the first set of sensor data.

7. The method of claim 6, further comprising instructing the display device to display the at least one polygon within the virtual environment.

8. The method of claim 1, further comprising identifying the physical object within the first set of sensor data by defining a mask about a plurality of pixels defining the physical object, wherein the mask extends the plurality of pixels defining the physical object by a predetermined number of pixels.

9. The method of claim 1, further comprising placing the virtual representation of the at least one physical object at a virtual location corresponding to a physical location of the at least one physical object.

10. The method of claim 1, wherein identifying the at least one physical object from the first set of sensor data further comprises identifying a first plurality of pixels of the first set of sensor data as a user and a second plurality of pixels of the first set of sensor data as background data.

11. The method of claim 10, wherein the at least one physical object is defined within the first set of sensor data by the background data.

12. The method of claim 1, further comprising generating the virtual representation of the physical object only in response to detecting that a user has entered a defined area associated with the physical object.

13. A non-transitory computer-readable storage medium comprising instructions for implementing a virtual representation of a physical environment using a virtual reality environment, that, when executed by a processor, configure the processor for:

receiving a first set of sensor data from a sensor;

identifying at least one physical object within a field of view of the sensor from the first set of sensor data;

determining a physical position of the at least one physical object using at least the first set of sensor data;

generating a virtual representation of the at least one physical object based at least in part on the first set of sensor data;

generating a virtual environment comprising the virtual representation of the at least one physical object and a virtual representation of at least one virtual object, wherein the virtual environment comprises a set of coordinates which correspond to a physical area in which the at least one physical object is present, and wherein the physical object is positioned in the virtual environment at a coordinate location that maps to the physical position of the at least one physical object; and

sending instructions to a display device to facilitate display of at least a portion of the virtual environment.

14. The non-transitory computer-readable storage medium of claim 13, wherein the display device is a head-mounted display.

15. The non-transitory computer-readable storage medium, wherein the instructions further comprise further comprise:

storing the virtual representation of the at least one physical object in a set of cached object data;

receiving a second set of sensor data;

determining that the at least one physical object is not present in the second set of sensor data;

retrieving the virtual representation of the at least one physical object from the set of cached object data; and

sending instructions to a display device to display the retrieved virtual representation of the at least one physical object at a last known position of the at least one physical object.

16-24. -24. (canceled)

25. A system for implementing a virtual representation of a physical environment using a virtual reality environment, the system comprising:

sensor interface circuitry configured to;

receive a first set of sensor data from a sensor;

identify at least one physical object within a field of view of the sensor from the first set of sensor data; and

determine a physical position of the at least one physical object using at least the first set of sensor data; and

virtual environment state management circuitry configured to;

generate a virtual representation of the at least one physical object based at least in part on the first set of sensor data;

generate a virtual environment comprising the virtual representation of the at least one physical object and a virtual representation of at least one virtual object, wherein the virtual environment comprises a set of coordinates which correspond to a physical area in which the at least one physical object is present, and wherein the physical object is positioned in the virtual environment at a coordinate location that maps to the physical position of the at least one physical object; and

send instructions to a display device to facilitate display of at least a portion of the virtual environment.

26. The system of claim 25, wherein the display device is a head-mounted display.

27. The system of claim 25, wherein the sensor interface circuitry is further configured to:

receive a second set of sensor data; and

determine that the at least one physical object is not present in the second set of sensor data;

and wherein the virtual environment state management circuitry is further configured to;

store the virtual representation of the at least one physical object in a set of cached object data;

retrieve the virtual representation of the at least one physical object from the set of cached object data; and

send instructions to a display device to display the retrieved virtual representation of the at least one physical object at a last known position of the at least one physical object.

28. The system of claim 27, wherein the sensor interface circuitry is further configured to determine that the at least one physical object is at least partially occluded prior to retrieving the virtual representation of the physical object form the set of cached object data.

29. The system of claim 25, wherein the at least one physical object is a user.

30. The system of claim 25, wherein identifying the at least one physical object comprises:

providing the first set of sensor data to a graphics processing unit; and

using at least one shader executing on the graphics processing unit to generate at least one polygon corresponding to he at least one physical object from the first set of sensor data.

31. The system of claim 26, wherein the virtual environment state management circuitry is further configured to instruct the display device to display the at least one polygon within the virtual environment.

32. The system of claim 25, wherein the sensor interface circuitry is further configured to identify the physical object within the first set of sensor data by defining a mask about a plurality of pixels defining the physical object, wherein the mask extends the plurality of pixels defining the physical object by a predetermined number of pixels.

33. The system of claim 25, wherein the virtual environment state management circuitry is further configured to place the virtual representation of the at least one physical object at a virtual location corresponding to a physical location of the at least one physical object.

34. The system of claim 25, wherein identifying the at least one physical object from the first set of sensor data further comprises identifying a first plurality of pixels of the first set of sensor data as a user and a second plurality of pixels of the first set of sensor data as background data.

36. The system of claim 25, wherein the virtual environment state management circuitry is further configured to generate the virtual representation of the physical object only in response to detecting that a user has entered a defined area associated with the physical object.

35. (canceled)