System and method for saving tracked data in the game server for replay, review and training

• US 10,717,001 B2
• Filed: 07/24/2017
• Issued: 07/21/2020
• Est. Priority Date: 03/25/2016
• Status: Active Grant

• Current Assignee: Zero Latency Pty., Ltd.
• Original Assignee: Zero Latency Pty., Ltd.
• Inventors: Vandonkelaar, Scott
• Primary Examiner(s): Lim, Seng H
• Application Number:
  US15/658,331
• Publication Number:
  US 20170319956A1
• Time in Patent Office: 1,093 Days
• Field of Search: None
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/31 Communication aspects specific to video games, e.g. between several handheld game devices at close range
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/497 Partially or entirely replaying previous game actions
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A system and method for replaying the activity on request to individuals and to the group at large within a virtual reality (VR) arena for training and efficiency improvement purposes from within the VR system or outside is disclosed. The virtual reality (VR) system does real time tracking and response feedback to the players, using light markers and cameras connected to multiple slave-server systems that are controlled by a master-server. The hierarchical data collection system collects all the activity and tracking data from the arena. The master-server combines and correlates all the data collected from the VR arena and sends that to the game-server to be saved in a dedicated storage memory coupled to the game server. This data is then played back as VR play back on request to individuals or to all players and supervisory staff within or outside the VR arena for training and efficiency improvement purposes.

Claims

1. A system for operating a virtual reality environment in a virtual reality arena comprising:

a master server;

a plurality of slave tracking servers in communication with and controlled by the master server;

a game server communicatively coupled to the master server;

a plurality of color cameras communicatively coupled to each of the plurality of slave tracking servers and configured to capture an image of at least one space used in the virtual interactive scenario experience and a plurality of objects within the space;

a mobile controller communicatively coupled to the game server and associated with the individual participant and an equipment device in the virtual reality environment;

a plurality of colored-light emitting tracking devices, each of the plurality of colored-light emitting tracking devices attached to each of the plurality of objects, wherein the plurality color cameras are configured to track the plurality of colored-light emitting tracking devices in the virtual reality environment;

wherein the game server is configured to provide feedback to the mobile controller associated with the individual participant,

wherein the plurality of colored-light emitting tracking devices uniquely identify each of the plurality objects tracked by the plurality of color cameras, the plurality of tracked objects comprising different individual participants and different equipment participating in the virtual interactive scenario experience in the virtual reality environment in the virtual reality arena,

wherein the plurality of color cameras is configured to track positions of the plurality of colored-light emitting tracking devices attached to each of the plurality of objects,

wherein each of the plurality of slave tracking servers is configured to determine optical tracking data and analyze the optical tracking data to identify the plurality of colored-light emitting tracking devices attached to each of the plurality of objects,

wherein the master server receives the optical tracking data from the plurality of slave tracking servers and wherein the master server is configured to determine position and activity data, wherein position and activity data comprises positions and activities of the plurality of objects tracked by the plurality of color cameras in the virtual reality environment within the virtual reality arena,

wherein the master server is further configured to send the determined position and activity data to the game server, and

wherein the game server is further configured to change a color of at least one of the plurality of colored-light emitting tracking devices if the at least one of the plurality of colored-light emitting tracking devices is within a predefined distance of another one of the plurality of colored-light emitting tracking devices assigned the same color within the virtual reality environment; and

a storage system coupled to the game server for storing the position and activity data received from the master server,

wherein the game server is configured to provide real-time feedback information to the mobile controller associated with the individual participant based on the position and activity data,

wherein the game server is further configured to retrieve and play back a segment of the stored position and activity data that relate to a time period just prior to a temporarily inactive individual participant within the virtual reality arena through a wireless communication link in response to receiving a request from within the virtual reality arena from the temporarily inactive individual participant,

wherein the game server is further configured to receive a request to provide a play back of the activity of the individual participant and equipment in the virtual environment based on the stored position and activity data.

2. The system of claim 1, further comprising a backpack personal computer electronically coupled to at least one of a head-mounted device, the mobile controller or the equipment carried by the individual participant.

3. The system of claim 1, wherein the plurality of colored-light emitting tracking devices are selected from the group consisting of fluorescent light sources and infrared bulbs.

4. The system of claim 1, wherein each of the plurality colored-light emitting tracking devices is configured to display multiple colors.

5. The system of claim 1, wherein the play back is for activity review within the virtual reality arena solely by the temporarily inactive individual participant requesting the play back, while the individual participant is in the inactive state and while the activity in the virtual reality arena is continuing.

6. The system of claim 1, wherein the play back is for training.

22. The system of claim 1, wherein the system is configured to present a plurality of virtual interactive scenario experiences in a plurality of virtual reality environments within the virtual reality arena.

23. The system of claim 22, wherein the master server compiles and processes data regarding the plurality of virtual interactive scenario experiences.

27. The system of claim 1, wherein the plurality of colored-light emitting tracking devices comprises a marker illumination device.

28. The system of claim 1, wherein the virtual reality arena is a free roaming arena configured to provide the virtual interactive scenario experience as a three-dimensional experience to the individual participants.

7. A system for replay of activities in a virtual interactive scenario experience presented in the virtual reality environment within a virtual reality arena, the system comprising:

a master server;

a plurality of slave tracking servers in communication with and controlled by the master server;

a game server communicatively coupled to the master server;

a plurality of color cameras communicatively coupled to each of the plurality of slave tracking servers, wherein the plurality of color cameras are configured to observe a plurality of colored-light emitting tracking devices attached to a plurality of objects, wherein the plurality of color cameras are configured to track the plurality of colored-light emitting tracking devices in the virtual reality environment,

wherein the tracked positions of the plurality of colored-light emitting tracking devices are processed by at least one of the plurality of slave tracking servers to determine optical tracking data,

wherein each of the plurality of slave tracking servers is configured to analyze the optical tracking data to identify the plurality of colored-light emitting tracking devices,

wherein the master server receives the optical tracking data from the plurality of slave tracking servers, wherein the master server is configured to determine position and activity data, wherein position and activity data comprises positions and activities of the plurality of objects tracked by the plurality of color cameras in the virtual reality environment within the virtual reality arena,

wherein the master server further provides the determined position and activity data to the game server, and

wherein the game server is further configured to change a color of at least one of the plurality of colored-light emitting tracking devices if the at least one of the plurality of colored-light emitting tracking devices is within a predefined distance of another one of the plurality of colored-light emitting tracking devices assigned the same color within the virtual reality environment; and

a storage system coupled to the game server for storing the position and activity data received from the master server,

wherein the game server is configured to provide real-time feedback information to a mobile controller associated with an individual participant based on the position and activity data, and

wherein the game server is also configured to retrieve and play back the stored position and activity data within the virtual interactive scenario experience presented in the virtual reality environment for a requested time period in response to receiving a request from an authorized individual participant during an inactive state of the authorized individual participant.

8. The system of claim 7, wherein the authorized individual is within the virtual reality arena during the inactive state.

9. The system of claim 7, wherein the authorized individual is outside the virtual reality arena during the inactive state.

10. The system of claim 7, wherein the stored position and activity data is played back through a wireless communication link to the individual participants solely within the arena when the individual participant is temporarily inactive within the virtual interactive scenario experience and when other participants are active within the virtual interactive virtual scenario experience presented in the virtual reality environment within the virtual reality arena.

11. The system of claim 7, wherein the stored position and activity data within the virtual interactive scenario experience presented in the virtual reality environment is played back through a communication link to the authorized individual participant.

12. The system of claim 7, wherein the inactive state comprises a temporary break in the virtual interactive scenario experience activity in the virtual reality environment.

13. The system of claim 7, wherein the inactive state comprises completion of the virtual interactive scenario experience activity in the virtual reality environment.

14. The system of claim 7, wherein the play back is available to all individual participants in the virtual interactive scenario experience in the virtual reality environment.

15. The system of claim 7, wherein the play back is available outside the virtual reality environment.

24. The system of claim 7, wherein the master server determines position and activity of individual participants and equipment in a plurality of virtual interactive scenario experiences in a plurality of virtual reality environments within the virtual reality arena.

16. A system comprising:

a processor;

a memory coupled to the processor, the memory containing a machine readable medium comprising machine executable code having stored thereon instructions for operating the system, wherein the system is configured to execute the machine executable code to cause the system to;

assign a specific color selected from a plurality of distinct colors to each one of a plurality of colored-light emitting tracking devices, the plurality of colored-light emitting tracking devices attached to a plurality of objects;

identify the plurality of objects in a virtual reality environment in a virtual reality arena by tracking the position of the plurality of colored-light emitting tracking devices using a plurality of color cameras,

wherein the plurality of objects includes an individual participant, a mobile controller communicatively coupled to a game server and associated with the individual participant, and an equipment device in the virtual reality environment;

change the specific color assigned to at least one of the plurality of colored-light emitting tracking devices if the at least one of the plurality of colored-light emitting tracking devices is within a predefined distance of another one of the plurality of colored-light emitting tracking devices assigned the same specific color within the virtual reality environment;

determine optical tracking data based on the tracked positions of the plurality colored-light emitting tracking devices, wherein the optical tracking data is analyzed to determine position and activity data of the plurality of objects and to provide real-time feedback information to the mobile controller associated with the individual participant based on the position and activity data;

receive a request to provide a play back of the activity of the individual participant and equipment in the virtual environment based on the stored position and activity data;

extract data responsive to the replay request to provide the play back of the activity of the individual participant and equipment in the virtual environment, the extracted data including the position and activity data collected for the identified plurality of objects; and

transmit the extracted data to an authorized requestor.

17. The system of claim 16, wherein the replay request is received from at least one of the participants in the virtual reality environment, and wherein the data corresponding to the replay request is transmitted to the at least one of the participants that requested the replay.

18. The system of claim 16, wherein the replay request is received from an administrator of the virtual reality environment, and wherein the data corresponding to the replay request is transmitted to all participants in the virtual reality environment.

25. The system of claim 16, wherein the processor is further configured to identify the plurality of objects in a plurality of virtual reality environments supported within the virtual reality arena.

19. A computerized method comprising:

assigning a specific color selected from a plurality of distinct colors to each of a plurality of colored-light emitting tracking devices attached to a plurality of objects;

identifying the plurality objects in a virtual reality environment within a virtual reality arena by tracking the position of the plurality of colored-light emitting tracking devices using a plurality of color cameras,

wherein the plurality of objects includes an individual participant, a mobile controller communicatively coupled to a game server and associated with the individual participant, and an equipment device in the virtual reality environment;

changing the specific color assigned to at least one of the plurality of colored-light emitting tracking devices if the at least one of the plurality of colored-light emitting tracking devices is within a predefined distance of another one of the plurality of colored-light emitting tracking devices assigned the same specific color within the virtual reality environment;

determining optical tracking data and analyzing the optical tracking data to identify the plurality of colored-light emitting tracking devices to determine position and activity data for the identified plurality of objects and transmitting the optical tracking data to a master server,

receiving the optical tracking data from plurality of slave tracking servers;

determining position and activity data, wherein position and activity data comprises positions and activities of the plurality of objects tracked by the plurality of color cameras in the virtual reality environment within the virtual reality arena;

sending the position and activity data to a game server for providing real-time feedback information to the mobile controller associated with the individual participant based on the position and activity data;

storing the position and activity data in memory;

receiving a request to provide a play back of the activity of the individual participant and equipment in the virtual environment based on the stored position and activity data;

extracting data responsive to the replay request to provide the play back of the activity of the individual participant and equipment in the virtual environment, the extracted data including the position and activity data collected for the identified plurality of objects; and

transmitting the extracted data to an authorized requestor.

20. The computerized method of claim 19, wherein the replay request is received from at least one of the participants in the virtual reality environment, and wherein the data corresponding to the replay request is transmitted to the at least one of the participants that requested the replay.

21. The computerized method of claim 19, wherein the replay request is received from an administrator of the virtual reality environment, and wherein the data corresponding to the replay request is transmitted to all participants in the virtual reality environment.

26. The computerized method of claim 19, further comprising identifying the plurality of objects in a plurality of virtual reality environments supported within the virtual reality arena.