Systems and methods for operating a virtual reality environment using colored marker lights attached to game objects

• US 10,430,646 B2
• Filed: 01/26/2018
• Issued: 10/01/2019
• Est. Priority Date: 03/25/2016
• Status: Active Grant

• Current Assignee: Zero Latency Pty., Ltd.
• Original Assignee: Zero Latency Pty., Ltd.
• Inventors: Vandonkelaar, Scott
• Primary Examiner(s): Lim, Seng Heng
• Application Number:
  US15/881,497
• Publication Number:
  US 20180150686A1
• Time in Patent Office: 613 Days
• Field of Search: None
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/31 Communication aspects specific to video games, e.g. between several handheld game devices at close range
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: A63F 13/837 Shooting of targets
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 : involving reference images or patches
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/56 : relating to colour
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 17/002 : for television cameras
• CPC Class Codes: H04N 23/10 : for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2224 : related to virtual studio applications
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Systems and methods are disclosed for operating, calibrating, and dynamically optimizing a system for a virtual reality environment where colored marker lights are attached to objects, the objects including players, controllers, and devices related to the game. One or more color cameras are used to view one or more spaces, and track positions and orientations of players and other objects according to the attached marker lights. A hierarchical system of servers may be used to process positions and orientations of objects and provide controls as necessary for the system. A method for color assignment is described as well as a calibration process, and a dynamic optimization process. A synchronization process is also described that ensures that a plurality of cameras and attached servers are properly coordinated. Head-mounted devices may also be used in conjunction with marker lights to provide information regarding players.

Claims

1. A system for operating a virtual reality environment including at least one space, the system comprising:

at least one color camera configured to view the at least one space;

at least one object, wherein the at least one object comprises at least one of a player, a controller and a device related to the virtual reality;

at least one colored light attached to each of the at least one object configured to be used as a tracking marker;

at least one server configured to track movements of the at least one object during the virtual reality;

a memory containing machine readable medium comprising machine executable code having stored thereon instructions for tracking the movements of the at least one object; and

a control system comprising a processor or processors coupled to the memory, the control system configured to execute the machine executable code to cause the control system to;

assign a color choice to each of the at least one colored light, wherein the assignment of the first color to the first colored light from among the at least one colored light is based on a spatial proximity of the first colored light to other colored lights in the at least one space.

2. The system of claim 1, wherein, if a total number of distinct assignable colors is greater than or equal to a total number of the at least one colored light, the control system is further configured to assign each of the at least one colored light a distinct color from among the total number of distinct assignable colors.

3. The system of claim 1, wherein, if a total number of distinct assignable colors is smaller than a total number of the at least one colored light, the control system is further configured to:

assign a distinct color from among the total number of distinct assignable colors to a first set of the at least one colored light, and

assign a color to each colored light from among a second set of the at least one colored light based on spatial proximity of each colored light from among the second set of the at least one colored light to remaining colored lights with assigned colors.

4. The system of claim 1, wherein the color choice for each of the at least one colored lights is selected from the group consisting of Red, Green, and Blue and combinations thereof.

5. The system of claim 4, wherein the color choice for each of the at least one colored lights further comprises half intensity secondary and tertiary colors.

6. The system of claim 1, wherein the specified distance is 5 meters.

7. A method for operating a virtual reality environment including at least one space, at least one color camera, at least one object, at least one colored light attached to each of the at least one object, at least one processor, and at least one server, the method comprising:

viewing, using at least one of the at least one color camera, the at least one space;

assigning, using at least one of said at least one processor, a first color to a first colored light from among the at least one colored light, within the at least one space based on a spatial proximity of the first colored light to other colored lights in the at least one space; and

tracking, using at least one of said at least one server, movements of the at least one object;

wherein the at least one object comprises at least one of a player, a controller and a device related to the virtual reality.

8. The method of claim 7, wherein, if a total number of distinct assignable colors is greater than or equal to a total number of the at least one colored light, the assigning further comprises assigning each of the at least one colored light a distinct color from among the total number of distinct assignable colors.

9. The method of claim 7, wherein, if a total number of distinct assignable colors is smaller than a total number of the at least one colored light, the assigning further comprises:

assigning a distinct color from among the total number of distinct assignable colors to a first set of the at least one colored light, and

assigning a color to each colored light from among a second set of the at least one colored light based on spatial proximity of each colored light from among the second set of the at least one colored light to remaining colored lights with assigned colors.

10. The method of claim 7, wherein the color choice for each of the at least one colored lights is selected from the group consisting of Red, Green, and Blue and combinations thereof.

11. The method of claim 10, wherein the color choice for each of the at least one colored lights further comprises half intensity secondary and tertiary colors.

12. The method of claim 7, wherein the specified distance is 5 meters.

13. A system for operating a virtual reality environment including at least one space, the system comprising:

at least one color camera configured to view the at least one space;

at least one object, wherein the at least one object comprises at least one of a player, a controller and a device related to the virtual reality;

at least one colored light attached to each of the at least one object configured to be used as at least one tracking marker;

a communication link;

at least one server, comprising a hierarchy of at least one slave server and a master server, configured to track movements of the at least one object during the virtual reality;

a memory containing machine readable medium comprising machine executable code having stored thereon instructions for tracking the movements of the at least one object; and

a control system comprising a processor or processors coupled to the memory, the control system configured to execute the machine executable code to cause the control system to;

locate, using the at least one slave servers, positions of at least a first and second tracking marker within view of the at least one color camera;

transmit, via the communication link, a position of the at least one tracking marker to the master server;

based on the positions of at least the first and second tracking markers, creating a composite tracking marker corresponding to a least the first and second tracking markers.

14. A method for operating a virtual reality environment including at least one space, at least one color camera, a communication link, at least one object, at least one colored light attached to each of the at least one object configured to be used as at least one tracking marker, at least one processor, and at least one server comprising a hierarchy of at least one slave server and a master server, the method comprising:

viewing, using at least one of the at least one color camera, the at least one space;

tracking, using at least one of said at least one server, movements of the at least one object;

locating, using the at least one slave servers, the position of the at least one tracking marker within view of the at least one color camera;

transmitting, via the communication link, position of the at least one tracking marker to the master server; and

determining, based on the position of the at least one tracking marker as viewed by a plurality of color cameras, an average position for the at least one tracking marker;

wherein the at least one object comprises at least one of a player, a controller and a device related to the virtual reality.

15. A system for operating a virtual reality environment including at least one space, the system comprising:

at least one color camera configured to view the at least one space;

at least one object, wherein the at least one object comprises at least one of a player, a controller and a device related to the virtual reality;

at least one colored light attached to each of the at least one object configured to be used as at least one tracking marker;

at least one server configured to track movements of the at least one object during the virtual reality;

a memory containing machine readable medium comprising machine executable code having stored thereon instructions for tracking the movements of the at least one object; and

a control system comprising a processor or processors coupled to the memory, the control system configured to execute the machine executable code to cause the control system to;

detect a calibration object placed in a space visible to each of the at least one color camera, wherein the calibration object comprises a plurality of colored calibration marker lights, and wherein the calibration object is placed in the space in a specified orientation relative to the at least one colored camera;

determine in a captured image at each of the at least one color camera, a position of each of the calibration marker lights; and

determine best fit positions and orientations of the calibration marker lights respective to each of the at least one color camera for use thenceforth in operation of the virtual reality.

16. The system of claim 15, wherein the calibration object is “

L”

shaped and comprises three colored calibration marker lights.

17. The system of claim 16, wherein each leg of the “

L”

shape of the calibration object is 1 meter in length.

18. The system of claim 16, wherein the three colored calibration marker lights are colored green, blue, and red.

19. A method for operating a virtual reality environment including at least one space, at least one color camera, at least one object, at least one colored light attached to each of the at least one object, at least one processor, and at least one server, the method comprising:

viewing, using at least one of the at least one color camera, the at least one space;

tracking, using at least one of said at least one server, movements of the at least one object;

detecting, using at least one of said at least one processor, a calibration object placed in a space visible to each of the at least one color camera;

determining, using at least one of said at least one processor, in a captured image at each of the at least one color camera, a position of each of two or more calibration marker lights affixed to the calibration object; and

converting, using at least one of said at least one processor, a detected orientation of the calibration object into a position and orientation respective to at each of the at least one color camera for use thenceforth in operation of the virtual reality,

wherein the at least one object comprises at least one of a player, a controller and a device related to the virtual reality.

20. The method of claim 19, wherein the calibration object is “

L”

shaped and comprises three colored calibration marker lights.

21. The method of claim 20, wherein each leg of the “

L”

shape of the calibration object is 1 meter in length.

22. The method of claim 20, wherein the three colored calibration marker lights are colored green, blue, and red.